A General Representation-Based Approach to Multi-Source Domain Adaptation
Authors: Ignavier Ng
Yan Li
Zijian Li
Yujia Zheng
Guangyi Chen
Kun Zhang
Venue: ICML 2025 poster (2025)
Keywords: Domain adaptation
representation learning
Primary area: theory->domain_adaptation_and_transfer_learning

Abstract: A central problem in unsupervised domain adaptation is determining what to transfer from labeled source domains to an unlabeled target domain. To handle high-dimensional observations (e.g., images), a line of approaches use deep learning to learn latent representations of the observations, which facilitate knowledge transfer in the latent space. However, existing approaches often rely on restrictive assumptions to establish identifiability of the joint distribution in the target domain, such as independent latent variables or invariant label distributions, limiting their real-world applicability. In this work, we propose a general domain adaptation framework that learns compact latent representations to capture distribution shifts relative to the prediction task and address the fundamental question of what representations should be learned and transferred. Notably, we first demonstrate that learning representations based on all the predictive information, i.e., the label's Markov blanket in terms of the learned representations, is often underspecified in general settings. Instead, we show that, interestingly, general domain adaptation can be achieved by partitioning the representations of Markov blanket into those of the label's parents, children, and spouses. Moreover, its identifiability guarantee can be established. Building on these theoretical insights, we develop a practical, nonparametric approach for domain adaptation in a general setting, which can handle different types of distribution shifts.

Unsupervised domain adaptation aims to transfer knowl- edge from labeled source domains an unlabeled target domain, particularly in scenarios where the training and A General Representation-Basel Approach to Multi-Source Domain Adaptation
Ignavier Ng"
Yan Li Zijian Li
Yujia Zheng' Guangyi Chen 12 Kun Zhang 12
Abstract
testing data distributions differ substantially. In a multi- source domain adaptation (MSDA) setup, each source do- main u E ( M) provides access to labeled dataset (x("), y = {(x} 9/k=)) ) where my represents the number of samples in domain u. Here, the i-th dimension of the feature vector X is denoted as X,, and (w) corre- sponds to the value of the i-th feature for the k-th sample in domain u. The goal is to train a classifier that generalizes to an unlabeled target domain, where only the feature vectors x" {x[]==
A central problem in unsupervised domain adap- tation is determining what to transfer from labeled source domains to an unlabeled target domain. To handle high-dimensional observations (e.g., im- ages), line of approaches use deep learning to learn latent representations of the observations, which facilitate knowledge transfer in the latent space. However, existing approaches often rely on restrictive assumptions to establish identifiability of the joint distribution in the target domain, such as independent latent variables or invariant label distributions, limiting their real-world applicabil- ity. In this work, we propose general domain adaptation framework that learns compact latent representations to capture distribution shifts rela- tive to the prediction task and address the funda- mental question of what representations should be learned and transferred. Notably, we first demon- strate that learning representations based all the predictive information, i.e., the label's Markov blanket in terms of the learned representations. is often underspecified in general settings. In- stead, we show that, interestingly, general domain adaptation can be achieved by partitioning the representations of Markov blanket into those of the label's parents, children, and spouses. More- over, its identifiability guarantee be estab- lished. Building on these theoretical insights, we develop a practical, nonparametric approach for domain adaptation in a general setting, which can handle different types of distribution shifts.
k== are available.
Determining the joint distribution Px, y in the target domain based solely on the marginal distribution Px is a fundamen- tally underdetermined problem. In the absence of additional assumptions, there are infinitely many possible joint dis- tributions Px, that can align with the observed marginal distribution. Therefore, assumptions that connect the source and target domain distributions essential for identifying the target joint distribution. Common approaches impose constraints to ensure a degree of similarity across these dis- tributions. A widely adopted assumption is covariate shift (Pan & Yang, 2009), which asserts that the conditional dis- tribution Py|x remains consistent across domains while the marginal feature distribution P'x varies. Alternatively, other frameworks account for variations in Py assume that transformations between the source and target features are linear (Zhang et al., 2015), offering additional ways to model domain relationships.
To avoid restrictive parametric assumptions about the re- lationships between domains, the principle of minimal changes is often considered (Schölkopf al., 2012: Zhang et al., 2013). This perspective is particularly effective when analyzed through the lens of the data generating process. For instance, when the underlying process is Y + X, the con- ditional distributions Py and Pxjy can vary independently across domains. By factoring the joint distribution this way, domain shifts can be represented in a parsimonious and structured manner. Moreover, changes in Pxty are often constrained to lie low-dimensional manifold, further simplifying the problem (Stojanov et al., 2019). Advances in domain adaptation frameworks, particularly those leverag- ing multiple-domain data, have demonstrated the feasibility of uncovering the data-generating process and capturing these domain shifts (Huang et al., 2020; Zhang et al., 2020).
1. Introduction
"quual contribution 'Carnegic Mellon University Mohamed bin Zayed University of Artificial Intelligence.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A General Representation-Baae Based Approach to Multi-Source Domain Adaptation
With the increasing capabilities of deep learning, another prominent line of work leverages neural architectures to map high-ddmeensoona features into latent representation space, ensuring that the latent variables Z are marginally invariant across domains. This approach is motivated by efficiency: by working in a lower-dimensional latent space, it aligns with the principle of minimal changes, as it only models the essential domain shifts while discarding irrelevant variations. A classifier can then be trained the labeled source data to ensure that the latent space retains predictive information about the labels (Ben-David et al., 2010; Ganin & Lempit- sky, 2015; Zhao et al., 2018; Li et al., 2024a). While this strategy enables domain alignment in the latent space, the joint distributions Pz.y may still vary significantly across domains, potentially degrading performance in the target domain. To address this, several works employ generative models or disentanglement techniques for the latent repre- sentations (Cai et al., 201 9a; Lu al., 2021; Yin et al., 2025). However, these methods typically rely on assumptions on the data distributions such as exponential family, or lack guarantees of identifiability for the target joint distribution y or the learned representations, limiting their ability to recover the true data generating process. The lack of identifiability raises concerns about the trustworthiness and reliability of these approaches, particularly when applied to real-world scenarios involving complex domain shifts.
for domain adaptation in a general setting, which can handle different types of distribution shifts. Finally, we validate our framework on real-world datasets, demonstrating that it outperforms existing methods.
2. Related Works
2.1. Domain Adaptation
Domain adaptation (Patel et al., 2015; Wilson & Cook, 2020; Farahani et al., 2021) aims transfer knowledge from labeled source domains to an unlabeled target domain, such that the model can generalize to the target domain. A classical approach is to learn domain-invariant representa- tions (Ganin & Lempitsky, 2015; Bousmalis et al., 201 6), which are extracted by aligning the features across differ- ent domains. For instance, Long et al. (2017; 2018) ap- plied maximum mean pseudo-labels and kernel methods for domain alignment, while Tzeng al. (2014) adopt an adaptation layer and domain confusion loss to learn domain- invariant representations.
A different line of works rely on the assumption that condi- tional distributions P(Z) | Y) remain stable across domains, enabling the extraction of domain-invarian representations for each class (Chen et al., 2019b;a; Kang et al., 2020). For instance, Xie et al. (2018) minimize inter-class domain dis- crepancy, while Shu et al. (2018) constrains boundaries to avoid high-density regions via virtual adversarial domain adaptation. Target shift, where Py varies across domains, has also been widely studied (Zhang et al., 2013; Lipton et al., 2018; Wen et al., 2020; Garg et al., 2020; Roberts et al., 2022). For instance, Tachet des Combes et al. (2020) devel- oped theoretical guarantees for the transfer performance un- der generalized label shift, while Shui et al. (2021) proposed selecting relevant source domains based on conditional dis- tribution similarity.
Recent works by Kong et al. (2022) and Li et al. (2024b) have introduced theoretical frameworks that establish differ- ent types of identifiability results for latent representations in domain adaptation. They partition the latent space into different subspaces according to its connection with do- mains or labels. Although different types of identifiability results have been provided for identifying the latent repre- sentations and joint distribution in the target domain, these works often rely on restrictive assumptions such as indepen- dent latent variables or invariant label distributions, limiting their real-world applicability.
Recent works incorporate causality into domain adapta- tion (Kong et al., 2022: Magliacane et al., 2018; Teshima et al., 2020; Chen & Bühlmann, 2021; Gong al., 2016; Sto- janov et al., 2019). For instance, Zhang et al. (2013; 2015) investigated target shift, conditional shift, and generalized target shift by assuming independent change for P(Y') and P(X Y). Cai et al. (2019a) learned disentangled seman- tic representations by leveraging causal generation process, while Stojanov et al. (2021) showed that Comain-invarian features require domain knowledge, giving rise to their pro- posed domain-speciff adversarial networks. These methods typically require restrictive assumptions and are not able to identify the latent variables with theoretical guarantees.
In this work, we propose general domain adaptation frame- work that learns compact latent representations to capture distribution shifts relative to the prediction task and address the fundamental question of what representations should be learned and transferred. Notably, we first demonstrate that learning representations based on all the predictive in- formation, such as the label's Markov blanket in terms of the learned representations, is often underspecified for do- main adaptation in general settings. Instead, we show that, interestingly, general domain adaptation can be achieved by partitioning the representations of Markov blanket into those of the label's parents, children, and spouses. Accordingly, we establish identifiability of the joint distribution in the target domain, by learning low-dimensional representations of the changing distributions. Building on these theoretical insights, we develop practical, nonparametric framework
2.2. Identification of Latent Variables
The identifiability of latent variables remains a fundamental challenge, as they are generally unidentifiable without addi-
2

---

A General Representation-Basee Approach to Multi-Source Domain Adaptation
tional assumptions (Hyvärinen & Pajunen, 1999; Locatello et al., 2019). In the case of linear mapping from latent to observed variables-Kkown
independent component analysis (ICA)- -ddntiiiiiiiiii can be achieved by assum- ing non-Gaussian latent variables (Comon, 1994; Hyvarinen et al., 2002). However, relaxing the linearity assumption leads to the ill-posed problem of nonlinear ICA (Hyvärinen & Pajunen. 1999; Hyvärinen et al., 2023).
Z1
Zz
Zi
Zu
To address this, existing nonlinear ICA methods typically rely on sufficient variations in the latent variable distribution, often introduced through auxiliary variables such as time or domain indices (Hyvarinen & Morioka, 2016; 2017; Hy- varinen et al., 2019; Khemakhem et al., 2020). Alternative approaches constrain the mixing function, either by restrict- ing it to specific function classes (Hyvärinen & Pajunen, 1999; Taleb & Jutten, 1999; Gresele et al., 2021; Buchholz et al., 2022) or enforcing sparsity (Zheng et al., 2022).
Xi
Xi
X2
Figure I: An example of the generative process considered in our work. The feature vector X is generated from latent variables Z. which, along with the label Y, follow a struc- tural equation model. The causal mechanisms, governed by parameters e(") and o(") may shift across domains. Here, X and the domain index u are observable. Furthermore, the label Y is available in the source domains but remains unobserved in the target domain. For this example, we have Zmb = {Zz, Zz. Za}. Zps [Z2]. Zeh {Zz}, Zaps = [2,,. and 2h = (Zz). To illustrate these latent variables, consider an example from PACS benchmark (Li et al., 2017): Y represents whether it horse, while Z2 captures key defining features (e.g., horse's 's head or horse- shoes), and Z3 represents attributes influenced by the horse (e.g., a saddle). Meanwhile, Zi and Za can represent back- ground elements.
More recently, causal representation learning has extended beyond ICA by considering causally-related latent variables instead of independent ones (Schölkopf et al., 2021). Simi- lar to nonlinear ICA, many approaches in this area leverage sufficient variations in the latent variable distributions, typi- cally induced by interventions (Ahuja al., 2023; Squires et al., 2023; von Kügelgen et al., 2023; Jiang & Aragam, 2023; Zhang et al., 2023; Varici et al., 2023; Vancı et al., 2024a;b; Jin & Syrgkanis, 2023; Bing et al., 2024; Zhang et al., 2024), temporal data (Yao et al., 2022a;b; Lippe et al., 2022: 2023), or both (Lachapelle et al., 2022: 2024). Other approaches rely on counterfactual view (Brehmer et al., 2022), multi-view data (Yao et al., 2024; Xu et al., 2024), more supervision information (Yang et al., 2021; Shen et al., 2022: Liang et al., 2023), causal ordering prior (Kori et al., 2023), constraint on the latent support (Ahuja et al., 2023;
Here, P)(Z;9) and PA(Y; represent the parents of Z and Y. respectively, in the DAG G. The e are mutually independent exogenous noise variables, and of=)) denotes the effective parameters (or latent factors) associated with each structural equation in the u-th domain. The generative process of each latent variable 24 may vary across domains, with the variation being determined by the corresponding pa- rameters g(()) Such variability is common in practice, e.g., arising from heterogeneous datasets, where the causal mech- anisms may shift. An example of the generative process depicted in Figure
Wang & Jordan, 2021), or structural constraints (Silva et al., 2006; Xie et al., 2020; Cai et al., 2019b; Xie et al., 2022:
Adams et al., 2021; Huang et al., 2022; Dong et al., 2023:
Kivva et al., 2021).
3. A Generative Model with Distribution Shift
We assume that the d-dimensional feature vector X (e.g., image pixels) is generated from latent variables Z = (Z,...., Zn) via an unknown, smooth, and invertible mix- ing function g R" - Rdd, Also, the label Y a categorical value that takes values from U....., ee. In each domain, the latent variables Z and the label Y are governed by structural equation model (SEM) that shares the same but un- known directed acyclic graph (DAG) G. The data-generating process can be summarized as follows:
Let Px,y Y; (X, and Pz,y (Z,Y; represent the joint distributions of X,Y and Z, Y. respectively, in the w-th domain. When the context is clear, we omit the sub- script for simplicity, and write p(u) (X, Y) and p(w) (Z, Y), respectively. We also assume that Pz,y and G satisfy the faithfulness assumption (Spirtes et al., 2001), and that Pz is third-order differentiable and positive everywhere on R"
Furthermore, we denote by Zmb, Zpa- Zen and Zape the Markov blanket', parents, children, and spouses of label Y, respectively. Also, let 2 denote the remaining latent
(Mixing)
X = g(Z),
Z, = e( )()), ie [n].
(1)
(SEM)
'In this work, we use the term "Markov blanket" to refer to the parents, children, and spouses a target variable.
Y = (x(P
3

---

A General Representation- Approach to Multi-Source Domain Adaptation
variables outside the Markov blanket of Y. and M be the Markov network over latent variables Z and label Y, whose edges are denoted by E.M). We define Omb- = (0:) z, & Zib and similarly for 0pa. Oeh and Spps- We also denote by Z, G. and M the learned latent variables, learned DAG, and learned Markov networks, respectively.
blanket Zub. ensuring that Zmb contains all and only the in- formation in Zmb- If such a representation can be recovered, we say that Zmb is subspace identifiable. With such a rep- resentation, the label Y becomes conditionally independent of all other variables Zlub. given 7mb- This implies that 2mb captures all essential information required to predict Y. Notably, this approach aligns with the feature selection literature (Yu et al., 2020), where the Markov blanket is rec- ognized as the minimal predictive set for the target variable.
4. Identifiability Theory
We present the identifiability theory for domain adaptation in a universal setting, where changes are allowed to occur anywhere in the latent space without restrictions. It is worth noting that specific types of domain shifts can be captured by imposing constraints on where changes occur. For instance:
However, recovering such a Markov blanket representa- tion Zmb is challenging without additional assumptions, as latent variable modeling often admits many spurious solu- tions (Hyvärinen & Pajunen, 1999; I Locatello et al., 2019). Fortunately, access to multi-domain data makes this recov- ery feasible. To achieve this, we rely on specific assumptions that require the distribution of latent variables to vary suffi- ciently across the source domains, formally described below.
Restricting changes to the parents of Y can be viewed as the covariate shift setting (Shimodaira, 2000).
Restricting changes to Y itself can be viewed as the target shift (Zhang et al., 2013) or prior probability shift (Storkey, 2009) problem.
Assumption 1 (Sufficient changes for 2). For each value of Z. there exist 2n + [M] +1 values of u, i.e., uk with ls 0, o....; .n + [M] such that the vectors w (Z, u&) - w ZZ un) with k: = 1, .............................. 2n |M] are linearly independent, where
Restricting changes to the children of Y can be viewed as the conditional shift (Zhang et al., 2013) problem.
vector w (Z, u) is defined
Oog ((
w(Z, u)
In contrast, our work considers the most general scenario, where changes may occur anywhere in the latent space, without imposing any specific restrictions.
iE(N]
8 log p(u) plu)(Z,)) (2,Y) 2
OZ? 0 log p(u) (Z,Y)
[E(n]
In Section 1, we discuss how learning latent representa- tions of the label's Markov blanket enables adaptation to certain types of domain shifts, while highlighting why this approach is often insufficient for domain adaptation. We then propose an alternative approach in Section 4.2 that in- volves learning latent representations of the label's parents, children, and spouses. Finally, in Section 4.3, we provide the identifiability guarantee for this approach.
0z,0Z;
22, JEE(M),
Assumption 2 (Sufficient changes for Y). For each value of Z, there exist //mbl + 1 values of (u, c) such that the vectors T (Z, ug, -c)--((22) Uk. (1) with or & e1 are linearly independent, where vector 7(2, c, is defined as
(Olog plw) p())(2) Y = ve) OZ
4.1. Subspace Identifiability of Latent Representations for Label's Markov Blanket
T(Z, c) c)
2.E2ub
With the advent of deep learning and its widespread adop- tion, many approaches leverage deep learning to learn com- pact latent representations of observations (Ganin & Lem- pitsky, 2015). These representations facilitate knowledge transfer in the latent space, enabling more efficient and effective transfer. The critical goal is then to learn latent rep- resentations that retain predictive information about Y. A traditional view is that the Markov blanket contains all infor- mation sufficient for predicting the target variable. Building on this perspective, a natural approach is to learn represen- tations that correspond to the Markov blanket of the label Y.
It is worth noting that different forms of sufficient change conditions have been adopted in nonlinear ICA (Hyvärinen et al., 2023) and causal representation learning (Schölkopf et al., 2021). These distribution changes, along with the invariant mixing function, offer valuable information for inferring the latent variables and their relations. We now provide identifiability theory to learn the latent representa- tions for the label's Markov blanket. The proof is provided in Appendix A and is inspired by Zhang et al. (2024). Al- though we state the faithfulness assumption (Spirtes et al., 2001) in the theorem above and Theorem 2, it suffices to adopt the single adjacency- faithfulnes (SAF) and single unshielded-collider faithulless (SUCF) assumptions (Ng et al., 2021; Zhang et al., 2024). These assumptions are
More specifically, the aim is to learn a representation Zmb that an invertible transformation of the label's Markov
4

---

A General Representation- Approach to Multi-Source Domain Adaptation
considerably weaker than the faithfulness assumption and ensure that the Markov network M is the same the moral- ized graph of the DAG G (Zhang et al., 2024, Proposition 2).
ing latent representations corresponding to three distinct subspaces of the label's Markov blanket: its parents, chil- dren, and spouses. Conceptually, this can be viewed as partitioning the Markov blanket into these three subspaces and focusing on recovering each subspace separately. In Section 4.3, we will show how such representations enable domain adaptation with identifiability guarantee in a univer- sal setting.
Theorem 1 (Subspace identifiability of Markov blanket). Consider the generative process in Equation (1). Suppose that Assumptions I and 2. as well as the faithfulness as- sumption, hold. By modeling the same generative process with minimal number of edges for the learned Markov net- work M, the learned Markov blanket Zub is an invertible transformation of the true Markov blanket Zmb.
Before presenting the assumptions and identifiability theory, we first introduce the notion of an intimate neighbor. Specif- ically, a latent variable Z; is said to be an intimate neighbor of Zj if Z, is adjacent to Zj and to all other neighbors of Z in M. Based on this, we introduce the following structural assumption on the latent DAG g:
However, learning latent representations that correspond to the subspace of the label's Markov blanket is insuffi- cient for domain adaptation in many scenarios. For in- stance, consider the factorization of the joint distribution P(Zmb, Y) = P(Y ((n))((Zm)))) If P(Zmb) changes across domains while P(Y Zmbb remains invariant, do- main adaptation can be achieved by using the same classi- fier (with Zmb or Zub as input) trained on the source do- mains in the target domain. This corresponds to a scenario where the conditional distributions P(Zpa (Zai 9)) or P(Zape PAAZappe G)) change across domains, while P(Y Zea.) and P(Zch/ PA(Zeh: remain invariant, which is clearly restrictive.
Assumption 3 (Group-specifie intimate neighbors). The intimate neighbors of label Y's parents, children, and spouses can only have intimate wsself their respective groups, i.e., other parents, children, or spouses Y
Y
This assumption is rather mild as it permits edges among the parents, children, and spouses of Y, but restricts edges between intimate neighbors belonging to different groups. In practice, intimate neighbors may be relatively rare. This assumption is necessary because, without additional con- ditions, it is generally not possible to disentangle Z₁ from Z, if Z is intimate neighbor of Zz. as supported by the theory in Zhang et al. (2024).
Now consider an alternative scenario where the factoriza- tion is given by P(Zub: Y)= P(zmb | PY).,. where P P(Zmbl Y) remains invariant across domains while P(Y) changes. This is known as the target shift (Zhang et al., 2013) or prior probability shift (Storkey, 2009) problem. However, with subspace identifiability of the label's ' Markov blanket indicated by Theorem 1, we do not know which parts of the learned representations correspond to the label's chil- dren, spouses, or parents. In this case, one may also factor- ize the distribution as P(Zmb, Y)= P(Y Zmm)) where both conditional distributions are allowed to change. Since Y is not available in the target domain and P(Y Zmb) changes, one no longer has identifiability of distribu- tion P(Zmb: Y) in the target domain.
Next, we present the identifiability theory for learning rep- resentations of the subspaces corresponding to parents, chil- dren, and spouses. The proof is given in Appendix B. Theorem 2 (Subspace identifiability of parents, children, and spouses). Consider the generative process in Equa- tion (1). Suppose that Assumptions 1, 2 and 3, as well as the faithfulness assumption, hold. By modeling the same gener- ative process with minimal number of edges for the learned Markov network M, there exists partition of the learned Markov blanket Zinb denoted as Zg,, Zsz and Zgg. such that they are invertible transformations of the true parents Zpa children Zeh, and spouses Zops respectively.
This motivates us to separate the representations of Zmb into three different subspaces in the next subsection, allow- ing us to improve the identifiability and to have more parsimonious representation of the changes.
The above theorem implies that the latent representations of parents, children, and spouses can be disentangled, allowing for the recovery of their respective subspaces. In the next subsection, we explain how these representations facilitate domain adaptation in universal setting with identifiability guarantee.
4.2. Subspace Identifiability of Latent Representations for Label's Parents, Children, and Spouses
In the previous subsection, demonstrated that learning latent representations corresponding to the subspace of the label's Markov blanket is often insufficient for domain adap- tation. This limitation arises, in part, because such represen- tations are overly coarse- -garieed To address this issue, we propose more fine-graine -ne-ggai approach that involves learn-
4.3. Identifiability of Joint Distribution in Target Domain
Building on the identifiability of latent representations es- tablished in Section 4.2, we now demonstrate how this facil- itates domain adaptation with identifiability guarantees in a
5

---

A General Representation-Bane Based Approach to Multi-Source Domain Adaptation
general setting. Specifically, the objective is to identify the joint distribution Pr (X, Y) in the unlabeled target domain, or equivalenty, PT | X). since (T(X) is already known in the target domain.
Assumption 4 (Low-dimensional changes). For each value of Ve. the conditional distribution P(Zch Y = Ve: Zeaps) contains only finite number of identifiable parameters that vary across domains. Furthermore. there is sufficiently large number of source domains.
To relate the conditional distribution Pr (Y X) at the level of raw observations X to the latent representations, we first state the following proposition and provide the proof in Appendix C.I.
Similar to Stojanov al. (2019), Assumption 4 im- plies the existence of bijective transformation h P P + R2, where denotes the dimensionality of the effective changing parameters. Under this trans- formation, the conditional distribution in each domain u can be expressed as a linear combination of the con- ditional distributions in the other source domains, i.c. h(P(()
Proposition 1. Consider the generative process in Equa- tion (1). We have
PT(Y- = ())
PT (Zch Y== = Uk. ) = Uk Ipa))
h( (Plu)
E. Li=ligeu Pie he Zon Zop
Zail Zai
P (zc Y = Uce Zops) PT (Y = V/c paa)
for some mixture weights ((a) ....., MMM Similarly, for the target domain T, there exist weights oier.... MM such that ) More
(u)
The above proposition implies that, to identify Pr PT | X). it suffices to identify PT (Zch Y = Ve, Zaps) and Pr (T Zpa) in the target domain. These conditional distributions are often simpler to model. However, the underlying latent variables Zpa: Zeh. and Zops are not directly observable and cannot be exactly recovered.
h(P2 Y= Ei= aT intuitively, Assumption indicates that all domain-speciffe conditional distributions (including source and target do- mains) for the label ve are confined to a g-dimensional manifold. Therefore, each conditional distribution for domain u can be characterized by the mixture weights (w) die ....,, Me' We denote the conditional distribution as- sociated with weights ae as pas (Zeh Y U/c Zsps).
Zea.l" uup
Fortunately, the identifiability theory developed in Sec- tion 4.2 shows that the subspaces corresponding latent variables Zpa. Zeh, and Zups can be disentangled. Specif- ically, if one can learn representations 2pa. Zeh. and Zaps that are invertible transformations of Zp Zeh. and Zuper respectively, then we have the following result.
We also adopt the following assumption, which ensures that the changes in conditional distributions are linearly indepen- dent. This is a rather mild assumption which requires that the conditional distribution varies sufficiently when their parameters change; otherwise, such parameter changes will not leave a sufficient footprint on the distribution shifts.
Corollary 1. Consider the generative process in Equa- tion (1), Let Zpa. Zeh, and Zaps be invertible transfor- mations of Zya Zen and Zapes respectively. We have
Assumption 5 (Linear independence). The elements in the set {BpPa (Zeh Y = Ve: Zapa) + 3' (Zeh Y = Ve, Zups): c 1, 1.....C C) are linearly independent for all Orey d/2 Ber Be 3. + 8% F O, if they are not zero.
PT = 0k 1X)
Pr (Zch Y Uk, PT Uk 2 P PT (Zeh Y Uee Zopa) Pr (Y- Uc paa)
With the assumptions above, we provide the identifiability result for the conditional distributions in the target domain.
The proof is available in Appendix C.2. From the corol- lary above, it suffices to identify the subspaces of the latent variables Zpa. Zeh. and Zaps up to invertible transforma- tions, e.g., by leveraging the identifiability theory developed in Section 4.2. Furthermore, the corollary implies that it suffices to establish the identifiability of the conditional dis- tributions pr (2chh Y, Zope) and PT (Y 2aa) in the target domain.
Theorem 3 (Identifiability of target distribution). Sup- pose that Assumptions 4 and 5 hold. Let 7 Zen and Zaps be invertible transformations of Z Zen and Zupes respectively. Suppose that we learn peew to match pr (Zen Zpp== Zaps) in the target domain, i.e., pnew (Zen I Zpan Zupps) = pr PrZZos I Zpar Zape) while constraining peew (Zoh Y, Zeps) to satisfy Assumption 4. Then, we have (Tcch Y, Zgpa)) -pnw (Zeh Yaga and PT (Y 2pa) = PY Zpa)
To ensure identifiability, we adopt the minimal change prin- ciple, which posits that the distributional changes across do- mains are confined to low-dimensional manifold (Stojanov et al., 2019). Specifically, we assume that the conditional distributions are governed by a small number of identifiable changing parameters, inspired by Stojanov et al. (2019). This enables us to identify these parameters by learning low- dimensional representations of the conditional distributions that vary across the source domains.
The proof is given in Appendix D and is inspired by Sto- janov et al. (2019). The core idea is that the learned low- dimensional representations allow us to reconstruct the con- ditional distribution in the target domain using unlabeled data in the target domain. Combined with the linear inde- pendence assumption, this further facilitates label prediction
6

---

A General Representation Beaeeeentation-aaeee Approach to Multi-Source Domain Adaptation
representation learning to characterize distributional shifts across domains, drawing inspiration partly from the frame- work presented by Zhang al. (2020). The approach oper- ates through principled multi-stage process grounded in identifiability theory and the necessity of isolating different components in the Markov blanket for accurate prediction of the target variable Y.
1(0.1)
LR
LKL,
Backbone Networks
18$1
Fz
7
Gz
Rspu
By
?
Zyps
Pen
First, we use variational autoencoders (VAEs) to match the distributions across source and target domains, extract- ing the required latent representations. Subsequently, we employ additional VAE (Kingma & Welling, 2014) mod- ules to explicitly model inter- variabl dependencies within the latent space, enabling systematic decomposition into block-level components while simultaneously estimating domain-specifte parameters 0. This design ensures that our framework effectively captures all variables constitut- ing the Markov blanket of Y, thereby facilitating robust cross-domain generalization and achieving accurate predic- tions in the target domain. Note that we use VAEs because they provide a convenient way to model the distribution of latent variables, and make it easier to incorporate prior structural information (e.g., parent-child relationships) into our method.
N(0.1)
Fy
Fzom
LRz
by
Zich
Lxiss
LR
1.oh
Gy
Gzen
Zen
22pc 0
P Zsp' Dcn
Figure 2: Overview of the General Approach for Multi- source Domain Adaptation (GAMA). The model first maps input images X to a latent space Z using VAE framework. The latent variables Z are partitioned into sev- eral components: Zmb. Zpa. Zeh. and Zaps. Two additional VAES are employed to capture the relationships among the latent variables and label, which aids estimating for improved predictions. For the three VAEs in total, we have the following losses: Lvae, LKI.1 + CRy and Ze. LKL, + LR.- Cross-entropy loss Ly and mean squared error (MSE) loss Leh are also used in the source domains to encourage better encoding. The final prediction is made by training& classifier on the inputs (Zpar Zapaz Zeh, 0 O()))
We now describe the specific model architecture. We first take an input image X and pass it through a backbone net- work (e.g., ResNet-50 (He et al., 2016)) to obtain a feature representation E. An encoder Fz then maps E into a latent space Z. Since we adopt a VAE framework (Kingma & Welling, 2014), a decoder Gz is also introduced recon- struct F. from Z. The reconstruction loss from the VAE enforces consistency between the original feature represen- tation E and its reconstructed version, preserving essential information. Here, we have the loss L = LKI, + LR, Note that La denotes reconstruction loss, while LKL de- notes Kullback-Leibler (KL) divergence; the index indicates loss for different VAEs. For example, CKL, denotes the KL divergence of the VAE from X to Z.
LR,
in the target domain. is worth noting that the result can be straightforwardy extended to multi-target domain adapta- tion by learning distinct pnew for each target domain.
Remark 1. In summary, one can first utilize Theorem 2 to learn demixing function ğ -i.e,, an encoder) that extracts latent representations of the label's 's parents, children, and spouses, up to certain indeterminacies. This same demixing function can then be applied to the target domain, where Theorem 3 guarantees the identifiability of the distributions pr (Zen Y, Zapa) and (T Zpa) in the target domain. Finally, applying Corollary I ensures the identifiability of pr X) in the target domain.
We partition the latent variable 2 as
= (Z Zpas Zen Zupa) E R."
According to Figure 1, we observe that, given Zmb, the elements relevant relevant to Y still include gf9) and g(v)) Once of*) and ((w) are accurately identified, combining them with Zmb yields a stable prediction (since all relevant information is then obtained).
5. Domain Adaptation Approach
Consider the data generation process involving 054) and glu).
Building on the theoretical insights established in the pre- vious section, we propose a General Approach for Multi- source domain Adaptation (GAMA) that systematically learns and identifies both latent variable structures and la- bel information in all domains. Our approach incorporates
(0 (1) Zpa) + Y and (0()) Y. Zsps)
+ Zeh.
In each domain, these f values are fixed parameters. Thus, we aim to learn 0 and Y so as to maximize P(Z () and

---

A General Representation-Bane Based Approach to Multi-Source Domain Adaptation
P(Z,Y (0) in the target domain. Formally, it is given by
then apply MLP-based VAËS alongside a classifier. Mean- while, for PACS, we employ ResNet18 as the backhone and similarly integrate MLP-based VAEs and a classifier. All metrics are computed by averaging over three random seeds.
E Yr (V ppa. log log p2 08y) YY)
max 0 ( A1 -2
- 81 KLL(1Q (Y pa. 0 (1 P(Y))
Baselines. To assess performance, we compare against several baselines, including the Source Only (He et al., 2016) approach and single-source domain adaptation meth- ods such as DAN (Long al., 2015), MCD (Saito et al., 2018), and DANN+BSP (Chen et al., 2019c). We further evaluate our model against leading multi-source domain adaptation techniques, including M3SDA (Peng et al., 2019), CMSS (Yang et al., 2020), LIC-MSDA (Wang et al., 2020), and T-SVDNet (Li et al., 2021). Additionally, we incorpo- rate comparisons with WADN (Shui al., 2021), which handles target shift in multi-source scenarios, well as iMSDA (Kong et al., 2022), recent framework that lever- ages component-wis wise identification for MSDA.
+ B Zanrugz (Zonl 2.pa. ). log P2 logp2
Zops: Y, ( Zeh)
- BI. Zech
Zupas Y. ) 1 (((cia)))
Two VAEs are used here. Specifically, ( 2pa) H Y involves an encoder Fy and a decoder Gy, while (0()) Y, Zspa) H Zeh involves an encoder Fze and a decoder Gza. We set 31 and 32 to 1. These lead to the losses Y and Zva..Z... Note that since Y is discrete, we cannot assume that P(Y) is Gaussian distribution (which is commonly done in VAE estimation), and thus use Gumbel-Softne VAE (Jang et al., 2017) which can convert the logit of Y into continuous variables for further calcu- lation. In the training stage, we treat Fy as the encoder producing Y. Since we have access to the ground truth labels Y in the source domains, we simply calculate the cross-entropy between Y and Y, giving rise to the losses Ly and L Lig-
Zen..
6.2. Numerical Results
The results for Office- and PACS datasets are provided in Table 1.
Office-Home dataset. GAMA achieves the best perfor- mance in most sub-tasks. On average, GAMA surpasses the strongest baseline (iMSDA) by a margin of 1%. This because our model tries to learn the pattern in the target do- main while training. By effectively predicting 0, our method is able to make more accurate inferences, leading to better performance.
Furthermore, in the source domains, since we have ac- cess to the ground truth Zeh we have the following MSE loss based on the encoded values Zeh to better capture the relationships between variables and the ground truth: Cch = (SE( Zeh), where MSE(-) denotes the mean squared error. Also, for the other VAE, we have Lvae,Ze. LKL, LR.
Finally, we can make the final prediction by using (Zpa Zaps: Zoh; of4), o(w)) with loss Lets- In conclu- sion, we have the following loss during training, where X1, X2 da.d4 and \s are hyperparameters:
PACS dataset. GAMA performs well for this dataset and achieves better accuracy than the best baseline on average. In particular, in the Photo domain, where the accuracy is already high, we have achieved an accuracy of 98.8%, which means that we have further explored the potential of the data.
fall = Lets + Xr + X2hnaee
+ >h&Caae Zer + Ashcn++ AsCr.
6.3. A Ablation Study
To evaluate the effectiveness of our special design to capture
6. Experiments
0 in the target domain, design two model variants: (1)
GAMA-vae: we remove all the VAE related losses; (2)
We show the effectiveness of our method compared with existing ones on widely used datasets in domain adapta- tion. Further details and empirical studies can be found in Appendix E.
GAMA-theta: we remove the losses brought by f-related
VABs: Y L.ve, Zet. we also remove Leh and Ly as some items for calculating these losses are related to B. Experiment results on the Office-Home dataset are shown in Table 2. It shows that the VAEs are essential for capturing information to perform adaptation. Moreover, with the O-related VAEs, one observes an improved accuracy.
6.1. Datasets and Baselines
Datasets. We validate our method on two well- known benchmarks for domain adaptation: Office- Home (Venkateswara et al., 2017) and PACS (Li et al., 2017). In each dataset, a single domain is designated as the target, and the remaining domains serve as sources. For Office- Home, we extract features using a pretrained ResNet50,
7. Conclusion
We develop a general, representation based domain adapta- tion framework that can handle different types of distribution
8

---

A General Representation-Bale Based Approach to Multi-Source Domain Adaptation
Table I: Results on Office-Home (Ar, CI, Pr, Rw) and PACS (P, A, C. S). A dash indicates no reported result. Baseline
results are taken from Kong et al. (2022).
Office-Home
PACS
Method
Ar
C1
Pr
Rw
Avg
P
C
Avg
DAN (Long et al., 2015)
68.3
57.9
78.5
81.9
71.6
Source Only (He et al., 2016)
64.6
52.3
77.6
80.7
68.8
94.5
74.9
72.1
64.7
76.6
DANN (Ganin et al., 2016)
64.3
58.0
76.4
78.8
69.4
91.8
81.9
77.5
74.6
81.5
DCTN (Xu al., 2018)
66.9
61.8
79.2
77.8
71.4
MDAN (Zhao et al., 2018)
91.4
79.1
76.0
72.0
79.6
WBN (Mancini et al., 2018)
97.4
89.9
89.7
58.0
83.8
MCD (Saito et al., 2018)
67.8
59.9
79.2
80.9
72.0
96.4
88.7
88.9
73.9
87.0
DANN+BSP (Chen et al., 2019c)
66.1
61.0
78.1
79.9
71.3
M3SDA (Peng ct al., 2019)
66.2
58.6
79.5
81.4
71.4
97.3
89.3
89.9
76.7
88.3
CMSS (Yang et al., 2020)
96.9
88.6
90.4
82.0
89.5
LIC-MSDA (Wang al., 2020)
97.2
90.2
90.5
81.5
89.8
T-SVDNet (Li al., 2021)
98.5
90.4
90.6
85.5
91.3
GeNRT (Deng et al., 2023)
98.5
93.6
91.4
85.7
92.3
iLCC-LCS (Liu et al., 2022)
95.9
86.4
81.1
86.0
87.4
WADN (Shui et al., 2021)
75.2
61.0
83.5
84.4
76.1
CASR (Wang et al., 2023)
72.2
61.1
82.8
82.8
74.7
TFEN (Li al., 2023b)
72.2
62.9
81.7
83.5
75.1
SSD (Li al., 2023a)
72.5
64.5
81.2
83.2
75.4
MIAN-y (Park & Lee, 2021)
69.9
64.2
80.9
81.5
74.1
iMSDA (Kong et al., 2022)
75.4
61.4
83.5
84.5
76.2
98.5
93.8
92.5
89.2
93.5
GAMA (Ours)
76.6
62.6
84.9
84.9
77.3
98.8
93.7
92.8
89.3
93.7
Table 2: Ablation study on Office-Hone comparing GAMA, GAMA-vae, and GAMA-thett
edge across different domains. The goal is to make models more efficient, especially in situations where labeled data is scarce. Our work does not introduce new ethical risks, as it focuses purely on enhancing the ability Al systems to generalize across domains without affecting sensitive areas. This research aims advance the field of machine learning, making it more practical and effective.
Method
Ar
Cl
Pr
Rw
Avg
GAMA
76.6
62.6
84.9
84.9
77.3
GAMA-vae
74.9
60.5
83.4
84.8
75.9
GAMA-theta
75.3
61.7
83.4
84.8
76.0
Acknowledgments
shifts. Specifically, we show that learning subspace of the label's Markov blanket representations is often underspec- ified for domain adaptation in many scenarios. To achieve show that one should parti- tion the subspace of Markov blanket into the subspace of the label's parents, children, and spouses. We then establish identifiability of the joint distribution in the target domain. Our resulting method provides a practical solution to do- main adaptation in general settings and outperforms existing methods various benchmark datasets, highlighting its po- tential for broader applications. Future works include eval- uating the method on larger-scale datasets and extending its application to diverse tasks, such as video, speech, and text.
The authors would like to thank the reviewers for their help- ful comments. We would like to acknowledge the support from NSF Award No. 2229881, AI Institute for Societal De- cision Making (AI-SDM), the National Institutes of Health (NIH) under Contract ROIHL 59805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. IN acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) Post- graduate Scholarships - Doctoral program.
general domain adaptation,
10985,
References
Adams, J., Hansen, N., and Zhang, K. Identification of partially observed linear causal models: Graphical con- ditions for the non- non-gaussian and heterogeneous cases. Advances Neural Information Processing Systems, 34: 22822-22833, 2021.
Impact Statement
Our paper presents a method for improving unsupervised domain adaptation to help Al models better transfer knowl-

---

A General Representation-Bane Based Approach to Multi-Source Domain Adaptation
Ahuja, K., Mahajan, D., Wang. Y., and Bengio, Y. Inter- ventional causal representation learning. In International Conference on Machine Learning, 2023.
Comon, P. Independent component analysis new con- cept? Signal Processing, 36:287-314, 1994.
Deng, Z.,, Li, D., He, J., Song, Y.-Z., and Xiang, T. Genera- tive model based noise robust training for unsupervised domain adaptation. arXiv preprint arXiv:2303.0577 2023.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F,, and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Dong, X., Huang, B., Ng, I., Song, X., Zheng, Y., Jin, S., Legaspi, R., Spirtes, P., and Zhang, K. A versatile causal discovery framework to allow causally-related hidden variables. In The Twelfth International Conference on Learning Representations, 2023.
Bing, S., Ninad, U.. Wahl, J., and Runge, J. Identifying linearly-mixed causal representations from multi-node interventions. In Conference on Causal Learning and Reasoning, 2024.
Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. Domain separation networks, Advances in neural information processing systems, 29, 2016.
Farahani, A., Voghoei, S., Rasheed, K., and Arabnia, H. A brief review of domain adaptation. In Stahlbock, R., Weiss, G. M., Abou-Nasr, M., Yang, C.-Y., Arabnia, H. R., and Deligiannidis, L. (eds.), Advances in Data Sci- ence and Information Engineering, pp. 877-894, Cham, 2021. Springer International Publishing. ISBN 978-3- 030-71704-9.
Brehmer, J., De Haan, P., Lippe, P., and Cohen, S. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319- 38331, 2022.
Ganin, Y. and Lempitsky, V. Unsupervised domain adapta- tion by backpropagation. In International conference on machine learning, pp. 1 1180-1189. PMLR, 2015.
Buchholz, S., Besserve, M., and Schölkopf, B. Function classes for identifiable nonlinear independent component analysis. In Advances in Neural Information Processing Systems, 2022.
Ganin, Y., Ustinova, E.,, Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 1 17(59):1-35, 2016.
Cai, R., Li, Z., Wei, P.. Qiao, J., Zhang, K., and Hao, Z. Learning disentangled semantic representation for do- main adaptation. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, 2019a.
Garg, S., Wu, Y., Balakrishnan, S., and Lipton, Z. A uni- fied view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
Cai, R., Xie, F., Glymour, C., Hao, Z., and Zhang, K. Triad constraints for learning causal structure of latent variables. Advances in neural information processing systems, 32, 2019b.
Gong, M., Zhang, K., Liu, T., Tao, D., Glymour, C., and Schölkopf, B. Domain adaptation with conditional trans- ferable components. In International conference on ma- chine learning, pp. 2839-2848. PMLR, 2016.
Chen, C., Chen, Z., Jiang, B., and Jin, X. Joint domain alignment and discriminative feature learning for unsu- pervised deep domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3296-3303, 2019a.
Gresele, L., Von Kügelgen, J., Stimper, V., Schölkopf, B., and Besserve, M. Independent mechanism analysis, a new concept? Advances neural information processing systems, 34:28233 -22248, 2021.
Chen, C., Xie, W., Huang, W., Rong, Y., Ding, X., Huang, Y., Xu, T., and Huang, J. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 627-636, 2019b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770777777777 770,7778111 2016.
Huang, B., Zhang, K., Zhang, J., Ramsey, J., Sanchez- Romero, R., Glymour, C., and Schölkopf, B. Causal discovery from heteroganeoushoonsattonaayy data. Jour- nal of Machine Learning Research, 21(89): 2020.
Chen, X., Wang, S., Long, M., and Wang, J. Transferability vs. discriminability: Batch spectral penalization for ad- versarial domain adaptation. In International conference on machine learning, pp. 1081-1090. PMLR, 2019c.
Huang. B., Low, C. H., Xie, F., Glymour, C., and Zhang, K. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Pro- cessing Systems, 35:5549-555 -5554, 2022.
Chen, Y. and Bühlmann, P. Domain adaptation under struc- tural causal models. The Journal of Machine Learning Research, 22(1):1 (651):11866-111 11155,11935,, 2021.
10

---

A General Representation-Base Approach to Multi-Source Domain Adaptation
Hyvarinen, A. and Morioka, H. Unsupervised feature ex- traction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Kong, L., Xie, S., Yao, W., Zheng, Y., Chen, G., Stojanov, P., Akinwande, V., and Zhang, K. Partial disentanglement for domain adaptation. In International conference on machine learning, pp. 1 11455- 14722
1472. PMLR, 2022.
Hyvarinen, A. and Morioka, H. Nonlinear ica of temporally dependent stationary sources. In Artificial Intellignnce and Statistics, pp. 460-469. PMLR, 2017.
Kori, A., Sanchez, P., Vilouras, K., Glocker, B., and Tsaf- taris, S. A. A causal ordering prior for unsupervised rep- resentation learning. arXiv preprint arXiv:2307.05700 2023.
Hyvärinen, A. and Pajunen, P. Nonlinear independent com- ponent analysis: Existence and uniqueness results. Neural networks, 12(3):429-439 1999.
Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. L., Lacoste, A., and Lacoste-Julien, S. Disentangle- ment via mechanism sparsity regularization: A new prin- ciple for nonlinear ICA. Conference on Causal Learning and Reasoning, 2022.
Hyvarinen, A., Karhunen, J., and Oja, E. Independent component analysis. Studies in informatics and control, 11(2):25--07, 2002.
Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. L., Lacoste, A., and Lacoste-Julien, S. Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprini arXiv:2401. 2202244110499 2024.
Hyvarinen, A., Sasaki, H., and Turner, R. Nonlinear ICA us- ing auxiliary variables and generalized contrastive learn- ing. In International Conference on Artificial Intelligence and Statistics, 2019.
Hyvärinen, A., Khemakhem, I., and Morioka, H. Nonlinear independent component analysis for principled disentan- glement in unsupervised deep learning. Putterns, 4(10): 100844, 2023. ISSN 2666-3899.
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Jang, E., Gu, S., and Poole, B. Categorical reparameteriza- tion with gumbel-softmax, In International Conference on Learning Representations, 2017.
Li, K., Lu, J., Zuo, H., and Zhang, G. Multidomain adapta- tion with sample and source distillation. IEEE Transac- tions on Cybernetics, 54(4):2193-2200 2023a.
Jiang, Y. and Aragam, B. Learning nonparametric latent causal graphs with unknown interventions. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Li, K., Lu, J., Zuo, H., and Zhang, G. Multi-source domain adaptation handling inaccurate label spaces. Neurocom- puttin,, 197:127 2024a.
Jin, J. and Syrgkanis, V. Learning causal representations from general environments: Identifiability and intrinsic ambiguity. arXiv preprint frXiv:2311.12266 2023.
Li, R., Jia, X., He, J., Chen, S., and Hu, Q. T-svdnet: Explor- ing high-orde prototypical correlations for molti-source domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9991 10000, 2021.
Kang, G. Jiang, L., Wei, Y., Yang. Y., and Hauptmann, A. Contrastive adaptation network for single-and multi- source domain adaptation. IEEE transactions pat- tern analysis and machine intelligence, 44(4):1733-1844 2020.
Li, Y., Wang, S., Wang, B.,, Hao, Z., and Chai, H. Trans- ferable feature filtration network for multi-source do- main adaptation. Knowledge- Based Systems, 260:1 13, 2023b.
Khemakhem, I., Kingma, D.. Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: A unify- ing framework. In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020.
Li, Z.,, Cai, R., Chen, G., Sun, B., Hao, Z.,, and Zhang, Subspace identification for tion. Advances in Neural Information Processing Systems, 36, 2024b.
domain adapta-
Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Repre- sentations, 2014.
Liang, W., Kekić, A., von Küggelgen, J., Buchholz, S., Besserve, M., Gresele, L., and Schölkopf, B. Causal component analysis. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Kivva, B., Rajendran, G., Ravikumar, P., and Aragam, B. Learning latent causal graphs via mixture oracles. Ad-
vances in Neural Information Processing Systems, 34: 18087-18101, 2021.
Lin, J. Factorizing multivariate function classes. Advances neural information processing systems, 10, 1997.

---

A General Representation-Base Approach to Multi-Source Domain Adaptation
Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, S. CITRIS: Causal identifiability from temporal intervened sequences. In International Confer- ence on Machine Learning, 2022.
computer vision and paltern recognition, pp. 3771-3780, 2018.
Ng. I., Zheng, Y., Zhang, J., and Zhang, K. Reliable causal discovery with improved exact search and weaker assump- tions. In Advances in Neural Information Processing Systems, 2021.
Lippe, P., Magliacane, S., Löwe, S., Asano, M., Cohen, T., and Gavves, E. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2023.
Pan, and Yang, Q. A survey on transfer learning. IEEE Transactions knowledge and data engineering, 22(10): 1345-1359 2009.
Lipton, Z., Wang, Y.-.., and Smola, A. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122- 3130. PMLR, 2018.
Park, G. Y. and Lee, S. W. Information theeoretic regulariza- tion for multi-source domain adaptation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9214-9223, 2021.
Liu, Y., Zhang, Z., Gong, D., Gong, M., Huang, B., Hengel, A. v. d., Zhang, K., and Shi, J. Q. Identifiable latent causal content for domain adaptation under latent covariate shift. arXiv preprint arXiv:2208. /4161, 2022.
Patel, V. M., Gopalan, R., Li, R., and Chellappa, R. Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53-69, 2015. doi: 10.1 N0.1100MMP:201422377555
Locatello, F., Bauer, S., Lucie, M., Raetsch, G.. Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In International conference on machine learning, pp. 4114-4124, PMLR, 2019.
Peng, X., Bai, Q.. Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Long, M., Cao, Y., Wang, J., and Jordan, M. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97- 105. PMLR, 2015.
Roberts, M., Mani, P., Garg, S., and Lipton, Z. Unsupervised learning under latent label shift. Advances in Neural Information Processing Systems, 35:18763-18778, 2022.
Saito, K., Watanabe, K., Ushiku, Y., and Harada, T. Max- imum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3723 3732, 2018.
Long, M., Zhu, H., Wang, J., and Jordan, M. I. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208-2211 PMLR, 2017.
Long, M., CAO, Z.. Wang, J., and Jordan, M. Con- ditional adversarial domain adaptation. In Bengio. S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi, N., and Garnett, R. (eds.), Advances in Neural Informa- tion Processing Systems, volume 31. Curran Associates, Inc., 2018
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E.,, Zhang, K., and Mooij, J. On causal and anticausal learning. Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 459- 466, 2012.
Lu, C., Wu, Y., Hernández-Lobato, J. M., and Schölkopf, B. Invariant causal representation learning for out-of- distribution generalization. In International Conference on Learning Representations. 2021.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Towards causal representation learning. Proceedings of the IEEE, 109(5):
612-634, 2021.
Magliacane, S., Van Ommen, T., Claassen, T., Bongers, S., Versteeg, P., and Mooij, J. M. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems, 31, 2018.
Shen, X., Liu, F., Dong, H., Lian, Q., Chen, Z., and Zhang, T. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241):1-55, 2022.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log-likeldoodd function. Jour- nal of statistical planning and inference, 90(2):227-244, 2000.
Mancini, M., Porzi, L., Bulo, S. R., Caputo, B., and Ricci, E. Boosting domain adaptation by discovering latent domains. In Proceedings of the IEEE conference on
12

---

A General Representation-Bane Bassd Approach to Multi-Source Domain Adaptation
Shu, R., Bui, H. H., Narui, H., and Ermon, S. A DIRT- T approach to unsupervised domain adaptation. arXiv preprint 1xXi::802.00835. 2018.
Tzeng, E.,, Hoffman, J., Zhang, N., Saenko, K., and Darrell, T. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014.
Shui, C., Li, Z., Li, J., Gagné, C., Ling, C. C. and Wang, B. Aggregating from multiple target-shifted sources. In International Conference on Machine Learning, pp. 9638- 9648. PMLR, 2021.
Varici, B., Acarturk, E.,, Shanmugam, K., Kumar, A., and
Tajer, A. Score-based causal representation learning with interventions. arXiv preprint (rXiv:2301.082300 2023.
VancI, B., Acartürk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based causal representation learn- ing: Linear and general transformations. arXiv preprint arXiv:2402.00849) 2024a.
Silva, R., Scheines, R., Glymour, C., and Spirtes, Learning the structure of linear latent variable mod- cls. Journal of Machine Learning Research, 7(8):191- 246, 2006. URL http::/t///// jmlr. patps///pps///// silva0 wt.6aa..httm html.
VaricI, B., Acartürk, E... Shanmugam, K., and Tajer, A. Lin- ear causal representation learning from unknown multi- node interventions. arXiv preprint arXiv:2406.05937, 2024b.
Spirtes, P., Glymour; C., and Scheines, R. Causation, Pre- diction, and Search. MIT press, 2nd edition, 2001.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 2017.
Squires, C., Seigal, A., Bhate, S. S., and Uhler, C. Linear causal disentanglement via interventions. In International Conference on Machine Learning, 2023.
Stojanov, P., Gong, M., Carbonell, J., and Zhang, K. Data- driven approach to multiple- source domain adaptation. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pp. 3487-3496. PMLR, 2019.
von Kügelgen, J., Besserve, M., Wendong, L., Gresele, L., Kekić, A., Bareinboim, E., Blei, D., and Schölkopf, B. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, 2023.
Stojanov, P., Li, Z.,, Gong, M., Cai, R., Carbonell, J., and Zhang. K. Domain adaptation with invariant representa- tion learning: What transformations learn? Advances in Neural Information Processing Systems, 34:24791- 24803, 2021.
Wang, H., Xu, M., Ni, B., and Zhang, W. Learning to com- bine: Knowledge aggregation for multi-source domain adaptation. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pp. 727-744. Springer, 2020.
Storkey, A. When Training and Test Sets Are Differ- ent: Characterizing Learning Transfer, pp. 3-28. 01 2009. ISBN 9780262170055 9780262170055.007 .00.500000000000
Wang, S., Wang, B., Zhang, Z.,, Heidari, A. A., and Chen, H. Class-aware sample reweighting optimal transport for multi-source domain adaptation. Nearocomputing. 523: 213-223, 2023.
doi: 10.755 I/mipresss
Strang, G. Linear Algebra and Its Applications. Thomson, Brooks/Cole, Belmont, CA. 4th edition, 2006.
Wang. Y. and Jordan, M. I. Desiderata for representa- tion learning: A causal perspective. arXiv preprint mrXiv:2109.03795, 2021.
Strang, G. Introduction to Linear Algebra. Wellesley- Cambridge Press, 5th edition, 2016.
Wen, J., Greiner, R., and Schuurmans, D. Domain ag- gregation networks for multi-source domain adaptation. In International Conference on Machine Learning, pp.
Tachet des Combes, R., Zhao, H., Wang, Y.-X., and Gordon, G. J. Domain adaptation with conditional distribution matching and generalized label shift. Advances Neural Information Processing Systems, 33:19276-19288 2020.
10214-10224. PMLR, 2020.
Wilson, G. and Cook, D. A survey of unsupervised deep domain adaptation. ACM Trans. Iniell. Syst. Technol. 11 (5), July 2020. ISSN 2157-6904. doi: 10.1145/3400066. 100.011145444005 URL htw.h:tts////m /oi. 1t1p://./////111140000000000000
Taleb, A. and Jutten, C. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 47 (10):2807-2820. 1999.
Teshima, T., Sato, I., and Sugiyama, M. Few-shot domain adaptation by causal mechanism transfer. In Interna- tional Conference on Machine Learning, pp. 9458-9469 PMLR, 2020.
Xie, F., Cai, R., Huang, B., Glymour, C., Hao, Z., and Zhang, K. Generalized independent noise condition for estimating latent variable causal graphs. In Advances in Neural Information Processing Systems, 2020.
13

---

A General Representation-Base Based Approach to Multi-Source Domain Adaptation
Xie, F., Huang, B., Chen, Z.., He, Y., Geng, Z., and Zhang, K. Identification of linear non-gaussian latent hierarchi- cal structure. In International Conference on Machine Learning, pp. 143700244388 PMLR, 2022.
Zhang, J., Greenewald, K., Squires, C., Srivastava, A., Shan- mugam, K., and Uhler, C. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 2023.
Xie, S., Zheng, Z., Chen, L., and Chen, C. Learning se- mantic representations for unsupervised domain adapta- tion. In International conference on machine learning, pp. 5423 1433-532. PMLR, 2018.
Zhang, K., Schölkopf, B., Muandet, K., and Wang. Z. Do- main adaptation under target and conditional shift. In International conference machine learning, pp. 819- 827. Pmlr, 2013.
Xu, D., Yao, D., Lachapelle, S., Taslakian, P., von Kügelgen, J., Locatello, F., and Magliacane, S. A sparsity principle for partially observable causal representation learning. In International Conference on Machine Learning, 2024.
Zhang, K., Gong, M., and Scholkopf, B. Multi-souree domain adaptation: causal view. In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, pp. 3150-3157, 2015.
Xu, R., Chen, Z., Zuo, W., Yan, J., and Lin, L. Deep cocktail Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 2018.
Zhang, K., Gong, M., Stojanov, P., Huang, B., Liu, Q., and Glymour, C. Domain adaptation a problem of inference graphical models. Advances in neural information processing systems, 33:4965 2020.
network:
Zhang, K., Xie. S., Ng. I., and Zheng. Causal repre- sentation learning from multiple distributions: A general setting. In International Conference Machine Learn- ing, 2024.
Yang, L., Balaji, Y., Lim, S.-N., and Shrivastava, Curricu- lum manager for source selection in multi- domain adaptation. In Computer Vision-ECCV 2020: 16th Eu- ropeun Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 608-624. Springer, 2020.
Zhao, H., Zhang, S., Wu, G., Moura, J. M., Costeira, J. P., and Gordon, G. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.
Yang, M., Liu, F., Chen, Z.,, Shen, X., Hao, J., and Wang, CausalVAE: Disentangled representation learning via neural structural causal models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Zheng, Y., Ng, I., and Zhang, On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
Yao, D., Xu, D., Lachapelle, S., Magliacane, S., Taslakian, P., Martius, G., von Kügglggen, J., and Locatello, F. Multi- view causal representation learning with partial observ- ability. In International Conference on Learning Repre- sentations, 2024.
Zheng, Y., Ng. I., Fan, Y., and Zhang, K. Generalized pre- cision matrix for scalable estimation of nonparametric Markov networks. In The Eleventh International Confer- ence Learning Representations, 2023.
Yao, W., Chen, G., and Zhang, K. Temporally disentangled representation learning. In Advances in Neural Informa- tion Processing Systems, 2022a.
Yao, W. Sun, Y., Ho. A., Sun, C., and Zhang, K. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Represen- tations, 2022b.
Yin, N., Wang, H., Yu, Y., Gao, T., Dhurandhar, A., and Ji, Q. Integrating markov blanket discovery into causal representation learning for domain generalization. In Computer Vision ECCV 2024, 2025.
Yu, K., Guo, X., Liu, L., Li, J., Wang, H., Ling, Z., and Wu, X. Causality- based feature selection: Methods and evaluations. ACM Comput. Surv., 53(5), 2020. ISSN 0360-0300.
14

---

A General Representation-Bale Based Approach to Multi-Source Domain Adaptation
Supplementary Material
A. Proof of Theorem 1
To prove the following theorem, we begin by establishing several intermediate results that are useful. We first prove Proposition 2, which is used in the proof of Proposition Building upon these two propositions, we then prove Proposition 4. Using Propositions 3 and 4, we proceed to establish Proposition 5. With these results, are ready to prove the following theorem, leveraging Propositions 3 and 5. It is worth noting that the overall proof strategy is partly inspired by Zhang et al. (2024), while ours is considerably more complex as it involves the discrete target variable Y (which is observed in the source domains).
Theorem (Subspace identifiability of Markov blanket). Consider the generative process in Equation (1). Suppose that Assumptions I and 2, as well as the faithfulness assumption, hold. By modeling the same generative process with minimal number of edges for the learned Markov network M, the learned Markov blanket Zmb is an invertible transformation of the true Markov blanket Zmb-
Proof. Recall that 2 denotes the recovered latent variables, M denotes the recovered Markov network, and V denotes the intimate neighbors of Z, By Propositions 3 and 5, there exists a permutation π of 2, denoted as 2,, such that the following statements hold:
(a) Zat() is solely function of a subset of (Z,) U V Z
(b) Mr and M are identical.
By Statement (b), under the faithfulness assumption (specifically the SAF and SUCF assumptions), the moralized graphs of and G are identical (Zhang et al., 2024, Proposition 2). Therefore, we have Z, E Zmb if and only if 2m(e) E Zmb-
Now suppose Zx(0) E Emb, which, by above reasoning, implies Z, Zbb By Statement (a), 2n(1) is solely function of a subset of {Zt} U Zz Here, we aim to show V S Zmb- Suppose Z, E V z. By definition, Z; and Y are adjacent in the Markov network M, and thus Zj is also adjacent to in M (because Zj is an intimate neighbor of Z,). This implies Zj E Zmb. Therefore, we have {Z:} } U Z C Zmh. i.e., 2x(() is solely function of a subset of Zmb. Since this holds for every Ez(i) E Zmb. we conclude that 2mb is solely a function a subset of Zmb.
Clearly, we can apply the same reasoning above (and Lemma in the reverse direction show that Zub is solely function of a subset of 7mb- Since the transformation from Z to 2 is diffeomorphism, conclude that 7mb is invertible transformation of Zmb-
A.1. Proof of Proposition 2
While the proof for the following proposition is inspired by Zhang et al. (2024, Proposition 1), ours involves discrete target variable Y (that is observed in the source domains), which requires the usage of Zheng al. (2023, Theorem 2) to handle it.
Proposition 2. Consider the generative process in Equation (1). Suppose that Assumptions I and 2 hold. Let 2 and M be the recovered latent variables and the recovered Markov network, respectively. By modeling the same generative process. we have the following statements:
(a) For each Z; and each (2x, 2,3 $ E(M), we have
az, 02;
02, 02, = 0.
(b) For each {Z, 2,} E ((M) and each (2,, 2,} & (MM), we have
0z 07,
0.
02, 02,
15

---

A General Representation-Basee Approach to Multi-Source Domain Adaptation
(c) For each (2, Y) E ((M) and each (2 Y) & E(M), have
az:
Proof. By definition, have X (Z) and 7 =g- where ! and 9 diffeomorphisms. Thus, the transformation from Z to Z, denoted by v=!, is diffeomorphism. Also, we have Y Y. By the frange-of-varibbe formula, obtain
log P(2,8) = log P(Z,Y) + log det Jul-
The first-order derivative is
log P(Z,8) 8)
Olog P(Z,Y) az:
log det Jol
+
(2)
aZ:
az;
0Z,
Let Zg and Z, be latent variables that are not adjacent in the recovered Markov network M. The second-order derivative
w.r.t. 24 and 2 is then given by
2 log P(Z,Y) 07, 07
i log P(Z,Y)
8%
82 log det Jol
+
Oz,0Z,
824 022
OZ,
02402
02,02
log PIZ,Y) Y) OZ, OZi
Σ
0 log P(Z,Y)
[0Z] 00:
+ 02, ozj
+
+
074
02, 02p
a7,07z
02, 02,
82, 02,
21
(7, Z7)EEM))(
Bog P(Z,Y) 8
82 log det Jul 02,02,
07,
02.02.
In the derivation above, we leveraged the following property (Lin, 1997): if 2. and 2, are not adjacent in the Markov
network M, then they conditionally independent given the remaining variables, which implies 8 log P(2.V) Similarly, this also the case for Zj and Z---
0.
02.02:
Now consider the u, and un domains where 1,...,21 2n + [M], and take the difference between the equations that correspond to them:
82 log p(a.) (2,Y)
82 log p(un) (2,Y)
07; 87:
O2?
OZ?
a2, a2₁
8 log p(u.) (Z,Y)
82 log p(vo) ( Y)
J22 0Z,
07, 07,
Σ
+
+
02,02;
a2,07;
07, 07
07, 07,
KC (Z
① Olgg p(u.) Z,
0 log ((an) Y)
82
+
Σ
OZ,
Oz,
a2,a2i
We collect the coefficients of the partial derivative terms in the equation above to form vector, and consider the vectors for r = 1,.... 2n + I.M.. Assumption A2 implies that these 2n + [M] vectors are linearly independent. Therefore, for any [Z,, Z E E(M) and (2,, 2,) & E(M), the following equations hold:
07, 07,
0,
(3)
02, 02,
82, 02:
0Z, Oz,
0, 02, 82,
(4)
02, 02,
02
0.
87,021
16

---

A General Representation-Base Based Approach to Multi-Source Domain Adaptation
Equation (3) implies that Statement (a) holds. By way of contradiction for Statement (b), suppose
Jzj 02,
oz.
+ O,
(5)
021 022
02₁
which, with Equation (3), implies 82, O. Substituting it into Equation (4), we have 82, 82: aă, with Equation (5). Therefore, Equation (5) must not hold, i.e.,
0, which is contradictory
0%, 02,
02 oZ. 0,
02, 00
indicating that Statement (b) holds. It then remains to prove Statement (c).
Now suppose that 2 and y are not adjacent in the Markov network M. By Zheng et al. (2023, Theorem 2), for each Cy + C1, we have
b log P(2, = =ve)) Olog P(2, y =va,) =0.
07x
07x
With Equation (2), we obtain
i log P(u) (2,Y = vc.))
Olog P(a) (( = [/yg
87
0z,
00
Olog p(u) (Z,Y vcr)
Olog p(u)
(Z,Y
Uei
0Z:
az:
22:
07k
(HZ,Y)EE(M
where the second line the equation follows from the same property in Zheng et al. (2023, Theorem 2). Under Assumption 2, there exist| Zubbl such equations above, and the Zuubl vectors formed by collecting those coefficients are linearly independent. This implies that Statement (c) holds, i.e.,
07
0.
A.2. Proof of Proposition 3
The proof for the following proposition is similar to Zhang et al. (2024, Theorem 2).
Proposition (Identifiability of Markov network). Consider the generative process in Equation (1). Suppose that Assumptions I and 2 hold. By modeling the same generative process, the Markov network M is identifiable up to isomorphism.
Proof. Since the transformation from 2 to Z is diffeomorphism, there exists permutation such that the diagonal entries in the permuted Jacobian matrix of such transformation are nonzero (e.g., see Zhang et al. (2024, Lemma 2) or Strang (2006; 2016)), which indicates
az,
+ 0, i == 1, , n.
(6)
02,000
Let Zi and Z, be two latent variables that are adjacent in the true Markov network M, but 2z(0) and 2-G) are not adjacent in the recovered Markov network M. With Proposition we obtain
az, az, 0,
a2m(6) 02
which is contradictory with Equation (6). Now suppose Z, and Ý are adjacent in the true Markov network M, but 2,(9) and Y are not adjacent in the recovered Markov network M. With Proposition 2, we obtain
aZ, 0,
02
17

---

A General Representation- Based Approach to Multi-Source Domain Adaptation
which is contradictory with Equation (6). Thus, have proved that M, is super-graph of M, i.e., all edges in M present in .. Since apply sparsity constraint on M during estimation such that it has smallest number of edges, conclude that M and M must be isomorphic.
A.3. Proof of Proposition 4
Proposition 4. Consider the generative process in Equation (1). Suppose that Assumptions 1 and 2 hold. Let Z and M be the recovered latent variables and the recovered Markov network, respectively. By modeling the same generative process. we have the following statements:
(a) For each Zi and each (2x, 2 & ((M), Zi is a function of at most one of Zs and Zr.
(b) For each {Z,, 2z} E EMM) and each (2,, 2i} & E(M), at most one of Ze and Z3 a function of Zn and Zi.
(c) For each {Z,, Y) E E(M) and each (2,, Y} & (MM). Ze is not a function of Zxk
Sketch of proof. By Proposition 2, for each {2,, } E E(M) and each (2x, Y} & E(M), we have
az:
0,
a2x
which implies that Statement (c) holds. Furthermore, by Statements (a) and (b) of Proposition 2, as well as Proposition 3, the same proof strategy of Zhang et al. (2024, Theorem 1) involving Intermediate Value Theorem can be used to show that Statements (a) and (b) of this proposition hold.
A.4. Proof of Proposition 5
The proof here is partly inspired by Zhang et al. (2024, Theorem 3), while ours involves a discrete target variable (that is observed in the source domains).
Proposition 5 (Identifiability of latent variables). Consider the generative process in Equation (1), Suppose that Assump- tions and 2 hold. Let 2 be the recovered latent variables, and & z, be the intimate neighbors of Zz. By modeling the same generative process, there exists a permutation π of Z, denoted as 2z, such that Za(s) is solely a function of a subset of {Z.) Z..
Proof. We first prove the following lemma.
Lemma 1. There exists a permutation π of 2, denoted as 2n, such that Z is solely a function of a subset of 2z(i) U
(Zx() Zr 10 2
Using Proposition 3 and its proof, there exists permutation æ of 2, denoted 2,, such that the Markov networks M and M2 are identical, and that Z, is a function of (z)))
Suppose Zj is not adjacent to Z, in Markov network M. This implies that Zx(1) and Zngy are not adjacent in M. Using Proposition 4, Z; is a function of at most one of 2x(() and Zmg) Since Z; is function of Zn(e) by definition, Z; must not be a function of 2zg)
Now suppose that Z, is adjacent to Z,, but not adjacent to some other neighbor of Z, We consider the following two cases:
Case 1: Zj is not adjacent Za. while Z& is adjacent to Z. This implies that 2n( and Zx(H) are not adjacent in M. Using Proposition 4, at most one of Z; and Zy is function of 2ng) and Z Since Z, is a function of 2=(w) by definition, Zi cannot be a function of 2z())
Case 2: Zj is not adjacent to Y, while Y is adjacent to Z; This implies that 2n(g) and Y are not adjacent in M. Using Proposition 4, Zy cannot be a function of Zx(g)-
18

---

A General Representation-Base Approach to Multi-Source Domain Adaptation
Thus, have proved Lemma Suppose Z, $ (Z:) U UV 2. which, by Lemma implies that Ze cannot be function of (o)) l.e.,
az
0Z,
0.
o0 ir 82m(r)
Using Zhang et al. (2024, Proposition 3) w.r.t. a2,"
we conclude that
az
=0
02
ir
and therefore
a2
17
0%
0.
az,
0z
aZ
ir
ir
That is, Zx(s) must not be a function of Z,. This implies that 27(() is solely a function of subset of {2:) U Vz
B. Proof of Theorem 2
The proof of the following theorem shares similar spirit with that of Theorem 1.
Theorem 2 (Subspace identifiability of parents, children, and spouses). Consider the generative process in Equation (1). Suppose that Assumptions 1, 2 and 3, as well as the faithifulness assumption, hold. By modeling the same generative process with minimal number of edges for the learned Markov network M, there exists partition of the learned Markov blanket zub denoted as Zs.., Zzg. and Ss. such that they are invertible transformations of the true parents Zpa children Zeho and spouses Zapas respectively.
Proof. Recall that å denotes the recovered latent variables, M denotes the recovered Markov network, and z, denotes the intimate neighbors of 7g. By Propositions 3 and 5, there exists a permutation of Z, denoted as 2,,, such that the following statements hold:
(a) Zn(s) solely a function of a subset of {2:} U Zz.
(b) M₂ and M are identical.
By Statement (b), under the faithfulness assumption (specifically the SAF and SUCF assumptions), the moralized graphs of G and G are identical (Zhang et al., 2024, Proposition 2). Therefore, we have Z; 7mb if and only if 2.(() 7mb-
Consider partition of 2mb. denoted as 2s, Zs, and 2sg. where
Žs, = (Z+(k)) Ux Zpa), Żs, (Zm(k) Ze E Zehl.
and Es. = (ZT(k) Zz E Zaps
Now suppose 2z(1)= Żs., which, by definition, implies Z, E Zpa. By Statement (a). 2z()) is solely a function of a subset of {2,) V.. 2. Under Assumption 3, we have 112 C Zpa This implies { U. C Zpa. i.e., 2n10) is solely function of subset of Zpa. Since this holds for every -z(1) Zs.. conclude that 2s. is solely function of a subset of Zpa.
Clearly, can apply the same reasoning (and Lemma 1) in the reverse direction show that Zpa is solely function of a subset of Zs,- Since the transformation from Z to Z is a diffeomorphism, we conclude that 2s, is an invertible transformation of Zpa.
The same reasoning above can be used to show that Es, and Es, are invertible transformations of Zeh and Zeps respectively.
C. Proof of Proposition and Corollary
C.1. Proof of Proposition 1
Proposition 1. Consider the generative process in Equation (1). We have
pr (T Y = Uk Zopa (Y Ukk 2pa)
PT(Y Uk X)
Σ Pr (Zon Y = Ve, Zspa) PT(Y Vo Zpa)
19

---

A General Representation-Base Based Approach to Multi-Source Domain Adaptation
Proof. We have
PT (Y = 2k X) = PT Uz, X)
(T(X)
PT(Y- Uk ))
(Change-of-variables)
(T(
PT(Y = Ukk 2)
PT (Y Uk Zub: Z
= -P' = Uk mmb)
(:: Y 1 2 Zmbb
PT(Y = Uk, 7mmh)
(m(Zmb) (T = -Uk Zmbb
co0l PT(Y = Ves Lmb)
PT(Y- =uk: 7ap Zeaps: Zch))
D Pr (V Ve, Tepas T Tesps Zech)
(Z Y = Uk. Za. Zopa P
PT (Y = Uk
Zpa: PT (Zpa: Zops)
(
1=1 PT(Zel (Zeh Y = Uc Lps Zeps PT (Y = Ue Zpai )
PT (Zpar Zopa)
Pr (Zoh = Uk. Zaps) PT (Y = iik Zpa)
Ee PT (Zeh Y Vos Zaps) ) pr (Y = U/. Zpa)
In the last step, we use the conditional independence relations Zeh 1 Zpa Y, Zsps and Y 1 Zaps Zpa.
C.2. Proof of Corollary 1
Corollary 1. Consider the generative process in Equation (1). Let Zpar Zeh- and Zaps be invertible transformations of pa. 7ec and Zaps respectively. We have
(z Y Uk. Zsps) U/ 2pa)) Ec PT (Zoh Y = Vo Zopa) Pr PT(( (Y Vo 2pa)
PT UK X)
Proof. By Proposition 1 and the change-of-variables formula, we have
Pr (Zeh Y Zopa PT (--(YY-)) U/ Zpa)
PT(Y = X)
Σ ==
Pr (Zeh Y = Ver Zopa PT = vic Zpa)
pd (Za, -ovy m. P (V Npu
P. (Y =uk (
P (Zpa)
0 p pr
Zoga)) P (Y =v (Y=veiZpa) (=(Zpa)
(V
pr
20nd)
P PY p (2pa) ( =0.22pp)
P (Y= 2xpe)
pr (2n, Rog
pr ?opp
P P=(Zps)
(T(con Y= Uk, Depa) PT =1h (pp))
E (r(Zch = Vc, Zops) Pr(Y U/c ppa)
D. Proof of Theorem 3
The proof of the following theorem is partly inspired by Stojanov et al. (2019).
Theorem 3 (Identifiability of target distribution). Suppose that Assumptions 4 and 5 hold. Let Zpas Zer and Zegs be invertible transformations of Zpas Zec and Zypes respectively. Suppose that we learn pnew to match Pr (Zech 7par 2ngps)
20

---

A General Representation-Base Based Approach to Multi-Source Domain Adaptation
in the larget domain, i.e., pne Zeh Assumption 4. Then, we have pr (Zeh Y, Zsps) = pme (Zehe Y, Zspa) and PT(Y Zpa) = pnew (Y) Zpa))
PT (Zch ppa: Zape) while constraining pnew (eer 1,, Zapa) to satisfy
Proof. We first have
pr (Zcch /pas Pisps)= pnew (Zeh P:pas Pusps)
Pr (Zoh, Zpai Zaps) pnee (Zoh: Zpa Zaps)
PT (T Zapa)
pnew (Zpas Zspa)
By the thange-o--variables variables formula and further simplifying, have
Pd (Zec:: 7aa Zssps)
paew Tech; Zepa: Zrpps)
PT (Zpa- Zops)
pnew Zpa Zups)
Ee Pr (Zen, Apps Zssaa Y [ve)
2
C
pnew Zeh; Zepay Zupos Y =Ve)
Pr (Zpa. Zopa)
Pnew (Zpa: Zapa)
>e Pr Y = Ve, Zaps) PT
2pa) (Zpp Zaps))
2
pnew (Zeh Y Vc, Zapss pnod (Y = ppa)
Pr (=(Z 1pp Zupps)
prew (Z Zrspa) pnew ( Zupps)
C
Σ PT(Y = =vo PPp) Y Ves Zsps) = wne (Y = Vo
1 pRew (Zeh Y = Ves Zspa).
Applying Assumption for pr (Zch Y Vos Zaps) and pnew pnew (Zeh Y Uns Zapa), we obtain
PT te App))pr (Zch Y Vc: Zapss) Σ
pnew (Y vca Zpa) po (Zch Y Ue: Zaps
which implies
Σ PY = Ue Pa (Zeh Y Uo: Zspps) pnow (Y = Ur pa (Zeh
pa.
Uc: Zmpa)) =0 0.
By Assumption 5, we have
PT = v. Sp (Zeh) Y = Ve, Zops) - pnew (Y = Ue Zpa) pa (Zeh Y = Ve, Zapa) = 0,
(7)
which, by taking integral w.r.t. Zeh. indicates
PT = ve Zpa) pnew (Y = Zpa).
(8)
Plugging the above equation into Equation (7) yields
pot (Zeh Y = Ves Zspa) = pa. (Zen Y Ve: Zapa):
or, equivalently,
pnee (Zeh Y Vn, Zepa) P (Zch Y Uey Zupps)
(9)
Applying change-of-variables formula to Equations (9) and (8), we obtain
pr (Z(c Y, Zupx) = ..ew p2ee Y, Zugas and PT(Y 2pp) pneew (Y 2pa)
21

---

A General Representation-Bacee Approach to Multi-Source Domain Adaptation
E. Experimental Details, Analysis and More Experiments
Model details. Our proposed approach adopts a hierarchical VAE architecture with the following detailed module designs. The domain size is M and the number categories C. The primary VAE encoder consists of fully connected layer (backbone features - hidden dimension) with batch normalization and ReLU activation, followed by two linear projections to generate mean μ and log- variance log O² for the latent space Z I Rdo +dzz, +dZeps where dg. dzz..> dz.. and dz..- denote the dimensions we set for Z Zps. Zch. and spa. respectively. The decoder reconstructs features through a two-layer MLP (latent dimension + hidden dimension - backbone feature dimension) with batch normalization and ReLU, Domain-speeific embeddings are implemented as learnable embedding layers: oy E 12 and deh E R- where day and do... denote the dimensions in both the encoder and decoder, which operate on the concatenated vector (0y, pa) RR +dzpa to predict/reconstruet class distributions. Similarly, we use encoder and decoder with linear layers to handle (0ch, Y, Zspa) Rd +C+dz for Zen reconstruction. The final classifier is a two-layer MLP that processes concatenated features (Zpar Zeh Zops By 0ch)) through a hidden layer with ReLU activation to output class predictions. All backbone features undergo adaptive average pooling and flattening before processing.
for oy and fch. respectively. 'he auxiliary VAE modules use single linear layers
Computing resources and efficiency. We train our model using a NVIDIA A A100-SXM4400GB GPU. For the Office- dataset, the batch size is set to 32, and the model is trained for 70 epochs, which takes approximately 160 minutes. The peak memory usage is around 35 GB. The majority of the computational cost comes from the ResNet-50 backbone, as only add several lightweight MLP layers after it. For the PACS dataset, the batch size is set to 32, and the model is trained for 70 epochs, each epoch has 200 steps, which takes approximately 32 minutes. The peak memory usage is around 11 GB.
Visualization and standard deviation. We have conducted visualizations of the latent space of features and VAE, Specifically, the t-SNE visualizations of the learned features on the Clipart task from the Office-Home dataset are available in Figure 3, which demonstrate the effectiveness of our method at aligning the source and target domains while preserving discriminative structures. We also report the standard deviations for Office-Home and PACS datasets in Tables 3 and respectively. In particular, GAMA only achieves the highest average accuracy but also exhibits very low variance, demonstrating its stable performance across different subtasks.
Table 3: Office-Homeee dataset results (accuracy std).
Method
Ar
CI
Pr
Rw
Avg
DAN (Long et al., 2015)
0.5
0.7
78.5
8..9
71.6
Source Only (He et al., 2016)
6..6
5.6
7.26
8.87
68.8
DANN (Ganin et al., 2016)
6.6
58.0
7.5
7.5
69.4
DCTN (Xu et al., 2018)
6..9
61.8
7.6
7..
71.4
MCD (Saito et al., 2018)
0.48
59.9
7.6
80.9
72.0
DANN+BSP (Chen et al., 2019c)
0.3
61.0+004
7.3
7.19+0.1
71.3
M3SDA (Peng al., 2019)
0.5
58.6
0.5
8..4
71.4
iMSDA (Kong al., 2022)
7.94
6.74
0.2
8.45
76.2
GAMA (Ours)
76.6 16.6+0.1
0.6
8.
+.1
77.3
22

---

A General Representation-Bane Approach to Multi-Source Domain Adaptation
Table 4: PACS dataset results (accuracy = std).
Method
Art
Cartoon
Photo
Sketch
Avg
Source Only (He al., 2016)
74.9
72.1 0.75
94.5 0.58
64.7 1.53
76.6
DANN (Ganin al., 2016)
$1.9-1.13
77.5 1.26
91.8 1.21
74.6 1.03
81.5
MDAN (Zhao et al., 2018)
0.36
0.70
0.85
7.80
79.6
WBN (Mancini et al., 2018)
8929
89.7+
97.4 0.84
58.0
83.8
MCD (Saito et al., 2018)
8.07
88.91.555
9.44=0.42
73.9
87.0
M3SDA (Peng et al., 2019)
8.43
$9.911000
0.33
7.87
88.3
CMSS (Yang et al., 2020)
0.36
9.84+0080
96.9 0.27
82.0
89.5
iMSDA (Kong et al., 2022)
93.75
92.46
98.48 0.07
8.722
93.48
GAMA (Ours)
98.77
9.77
92.81 + 0.40
89.27 0.68
93.65
(a) Our method,
(b) The iMSDA method.
Figure 3: The t-SNE visualizations of the learned features on the CClipart task in the Office-Home dataset. Specifically, red points indicate learned features form the source domains, while blue points indicate learned features from the target domain.
Table 5: Hyperparameters for Office-Home (Ar, CI, Pr, Rw) and PACS (P, A, C. S) datasets.
Office-Home
PACS
Parameter
Ar
CI
Pr
Rw 6 10 4
P
A
C
A1
2 x 10 10-3
x 10 4
3 3 10
x 10
3 10 3
5 10
9 x 10 3
A2
4 x 10-1
2 10-4
3x 10-3
x 10-4
4 x 10-4
1 10-1
1 10
x 10-3
Aa
x 10-4
8 10-4
1x 10-3
4 10-3
10-4
2 10-3
2 10-4
7 10
5 1
6 10
7 1
1x 10-3
1x 10-3
2 10
4 10
5 10
2 10-3
4x 10-4
3x 10-1
4 10-
4 10
9 10
4 10-3
5 10-3
23


Reviews:
[{"review_id": "RkfR21yaXO", "reviewer": "Reviewer_eV3Q", "summary": "The paper proposes a multi-source domain adaptaion approach. It is a generative based approach which projects feature representation into a latent space using VAE. The latent space (Markov blanket) is then partitioned into the subspace of label’s parents, children and spouses. Next, two VAE are used to Z_pa and Z_sps to learn \\theta_Y and \\theta_ch. A classification loss is added to train on input labels.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time dedicated to reviewing our paper and the valuable comments. We have tried to address all the concerns in the following.\n\n**Q1:** \"More experiments would be better\" and \"Domainnet is a popular dataset for multi-source domain adaptation\"\n\n**A1:** Thank you for your insightful suggestion. Following your suggestion, we are currently conducting experiments for the DomainNet dataset. The dataset contains 6 different sub-tasks. Due to time constraints of the rebuttal period, some of these experiments are still ongoing, and we have finished the experiments for the Clipart sub-task. The results are provided in the table below, which indicate that our method outperforms all other baselines. Note that to save space for author's response, the references of the baselines, if not specified, can be found in our paper (e,g., Table 1). We will include the complete results for different sub-tasks in the revised manuscript. Hope this addresses your concern.\n\n| Method      | Clipart |\n|-------------|------|\n| Source Only    | 52.1 |\n| DANN    | 60.6 |\n| DCTN    | 48.6 |\n| MCD    | 54.3 |\n| M3SDA    | 58.6 |\n| CMSS    | 64.2 |\n| LtC-MSDA    | 63.1 |\n| ADDA [1]    | 47.5 |\n| ML_MSDA [2]    | 61.4 |\n| meta-MCD [3]    | 62.8 |\n| PFSA [4]    | 64.5 |\n| SSD [5]    | 67.2 |\n| **GAMA (Ours)** | **69.2** |\n\n**Q2:** \"std should be added to the results\"\n\n**A2:** Thanks for this suggestion. We will include the standard deviation for all datasets in the revised manuscript. Due to space constraints of author's response, let us provide the results with standard deviation for Office-Home dataset in the table below.\n\n| Method | Ar | Cl | Pr | Rw | Avg |\n|-------------|------|------|------|------|------|\n| DAN    | 68.3±0.5 | 57.9±0.7 | 78.5±0.1 | 81.9±0.4 | 71.6 |\n| Source Only    | 64.6±0.7 | 52.3±0.6 | 77.6±0.2 | 80.7±0.8 | 68.8 |\n| DANN    | 64.3±0.6 | 58.0±1.6 | 76.4±0.5 | 78.8±0.5 | 69.4 |\n| DCTN    | 66.9±0.6 | 61.8±0.5 | 79.2±0.6 | 77.8±0.6 | 71.4 |\n| MCD    | 67.8±0.4 | 59.9±0.6 | 79.2±0.6 | 80.9±0.2 | 72.0 |\n| DANN+BSP | 66.1±0.3 | 61.0±0.4 | 78.1±0.3 | 79.9±0.1 | 71.3 |\n| M3SDA    | 66.2±0.5 | 58.6±0.6 | 79.5±0.5 | 81.4±0.2 | 71.4 |\n| iMSDA    | 75.4 ± 0.9 | 61.4 ± 0.7 | 83.5 ± 0.2 | 84.5 ± 0.4 | 76.2 |\n| **GAMA (Ours)** | **76.6±0.1** | **62.6±0.6** | **84.9±0.1** | **84.9±0.1** | **77.3** |\n\n**Q3:** \"Limited analysis\" and \"Some experiments related to the latent space of features and VAE would be nice to visualize\"\n\n**A3:** We appreciate your helpful suggestion, which will aid our understanding of the method. In light of your suggestion, we have conducted visualizations of the latent space of features and VAE. Specifically, the t-SNE visualizations of the learned features on the Clipart task from the Office-Home dataset are available in Figure 6 in the anonymized repository: https://anonymous.4open.science/r/icml-rebuttal-gama/rebuttal.pdf, which demonstrate the effectiveness of our method at aligning the source and target domains while preserving discriminative structures. We will include the visualizations in the revised manuscript, and hope this addresses your concern.\n\n**Q4:** \"The approach has too many hyper-parameters. How and what these values are set to is not available in the paper.\"\n\n**A4:** Thanks for your comment. We followed existing work [6] and selected hyperparameters that lead to optimal performance for each task. Due to space constraints of author's response, the values of these hyperparameters are provided in Table 1 of the anonymized repository: https://anonymous.4open.science/r/icml-rebuttal-gama/rebuttal.pdf. We will include these details in the revised manuscript for clarity.\n\n**We want to thank the reviewer again for all the valuable feedback.**\n\n---\n**References:**\n\n[1] Tzeng et al., Adversarial discriminative domain adaptation. In CVPR, 2017.\n\n[2] Li et al., Mutual learning network for multi-source domain adaptation. arXiv preprint arXiv:2003.12944, 2020.\n\n[3] Li et al., Online meta-learning for multi-source and semi-supervised domain adaptation. In ECCV, 2020.\n\n[4] Fu et al., Partial feature selection and alignment for multi-source domain adaptation. In CVPR, 2021.\n\n[5] Li et al., Multidomain adaptation with sample and source distillation. IEEE Transactions on Cybernetics, 2023.\n\n[6] Kong et al., Partial disentanglement for domain adaptation. In ICML, 2022.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Glanced over them.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper presents an incremental solution to multi-source domain adaptation problem.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "Strengths:\n1. The method is novel and has a detailed theoretical background and explanation.\n2. Approach achieves good results on Office Home and PACS datasets.\n\n\nWeakness:\n1. More experiments would be better. Proposed approach is evaluated on two datasets (std should be added to the results). It is suggested to include at least 3 datasets for reliability. Domainnet is a popular dataset for multi-source domain adaptation. \n2. Limited analysis. Analysis performed on the approach is not enough. Some experiments related to the latent space of features and VAE would be nice to visualize. \n3. The approach has too many hyper-parameters. How and what these values are set to is not available in the paper.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBbSKD3cXq", "reviewer": "Reviewer_eX2F", "summary": "The manuscript presents a general representation-based approach for multi-source domain adaptation (GAMA). It aims to improve knowledge transfer across domains by leveraging theoretical identifiability results for latent variables and adapting a variational autoencoder (VAE)-based framework. The authors propose partitioning the Markov blanket into its parents, children, and spouses to enhance adaptation performance and validate their approach through theoretical guarantees and empirical evaluations on two benchmark datasets.\n\n## update after rebuttal\nThe authors' answers clarified my doubts, so I confirm the positive assessment of the manuscript.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's time and valuable comments, many of which will help improve the clarity of our paper. Our responses to these comments are given below.\n\n**Q1:** \"some assumptions, such as linear independence of certain distributions, may be restrictive and require further discussion\"\n\n**A1:** Thanks for this comment. Intuitively, the sufficient change assumption requires the existence of multiple environments where the causal mechanisms among latent variables change sufficiently. These distributional changes, combined with the invariant mixing function, provide information for recovering the latent variables and their causal relationships. Note that this type of sufficient change assumption has been commonly adopted in the literature of nonlinear ICA and causal representation learning to establish identifiability under various settings. We will include this discussion in the revision.\n\n**Q2:** \"Lack of discussion on computational efficiency\" and \"Are there any practical limitations regarding computational resources for training?\"\n\n**A2:** We train our model using a NVIDIA A100-SXM4-40GB GPU. For Office-Home dataset, the batch size is set to 32, and the model is trained for 70 epochs, which takes approximately 160 minutes. The peak memory usage is around 35 GB. The majority of the computational cost comes from the ResNet-50 backbone, as we only add several lightweight MLP layers after it. We will discuss this and report the running time for all datasets in the revision.\n\n**Q3:** \"No discussion on potential failure cases or limitations\"\n\n**A3:** A limitation, as discussed in Q2, is that our method may require a relatively long training time. However, we believe this trade-off is justified given the performance improvements it achieves. We will discuss this in the revision.\n\n**Q4:** \"Provide additional baselines from causal representation learning\"\n\n**A4:** There are only a few works from causal representation learning that address domain adaptation. The only work we are aware of is iMSDA (Kong et al., 2022), which we have included as a baseline. Please kindly let us know if there is other suitable causal representation learning baseline you think we could compare to.\n\n**Q5:** \"Clarify the generalizability of the method beyond the considered datasets\"\n\n**A5:** Our method is designed for general domain adaptation, effectively handling various types of distribution shifts by learning latent representations that are most relevant for adaptation and capturing distribution shifts through low-dimensional representations. The datasets we used—Office-Home and PACS—are widely adopted benchmarks that span diverse domains and distribution shifts, providing strong empirical support for its generalizability.\n\nTo further evaluate the generalizability, we have conducted additional experiments under more severe label shifts; see our response to Q1 for Reviewer d2cE. It is observed that our method continues to perform well, demonstrating its effectiveness even in more challenging settings.\n\n**Q6:** \"Page 7, line 372 first column. I think the authors would refer to Figure 2 instead of Figure 1\"\n\n**A6:** We will fix the typo in the revision.\n\n**Q7:** \"How does the approach handle highly imbalanced domain shifts?\"\n\n**A7:** Based on your question, we interpret \"highly imbalanced domain shifts\" as referring to large distribution shifts between domains. Under this interpretation, our approach remains applicable as long as the assumptions hold. Specifically, the more imbalanced the domain shifts are, the more domains may be required to satisfy our theoretical assumptions. We will incorporate this discussion in the revised manuscript. If you intended a different interpretation of \"imbalanced domain shifts,\" we would greatly appreciate if you could kindly let us know.\n\n**Q8:** \"Would the method still be effective with fewer source domains?\"\n\n**A8:** While our theoretical result assumes multiple domains, our method remains highly effective even with only a limited number of source domains. This is demonstrated by the strong performance in our experiments, where only a limited number (i.e., three) of source domains are available. We will include this discussion in the revision.\n\n**Q9:** \"Could the theoretical results be extended to other types of unsupervised domain adaptation problems?\"\n\n**A9:** Our theoretical results are already quite general, as they accommodate (1) various types of distribution shifts, where changes may occur anywhere in the latent space, and (2) arbitrary relations among latent variables. To address your question, while our results assume a single target domain, **it can be naturally extended to multi-target domain adaptation** by learning distinct $P_{new}$ for each target domain. This extension is feasible because our framework learns compact latent representations that capture distribution shifts relative to the prediction task. We will include a discussion of this extension in Section 4.3.", "claims_and_evidence": "The manuscript makes strong theoretical claims regarding identifiability and its impact on domain adaptation. The theoretical framework is supported by a series of assumptions and theorems, including results on subspace identifiability of the Markov blanket and its components​. Empirical results on benchmark datasets substantiate the proposed method, demonstrating its effectiveness compared to existing approaches.", "methods_and_evaluation_criteria": "The authors employ a combination of variational autoencoders and deep neural networks to extract and learn latent representations that are invariant to domain shifts. Evaluation is conducted on standard datasets such as PACS and Office-Home​, using accuracy as the primary metric. The paper also includes an ablation study to analyze the impact of different components of the model​.", "theoretical_claims": "The manuscript provides rigorous theoretical backing, introducing multiple theorems to establish the identifiability of the joint distribution in the target domain. However, some assumptions, such as linear independence of certain distributions, may be restrictive and require further discussion​.", "experimental_designs_or_analyses": "The experiments are well-structured, covering multiple datasets and baseline comparisons. However, additional baselines, particularly recent causal representation learning approaches, could further validate the method's effectiveness​.", "relation_to_broader_scientific_literature": "The paper contextualizes its contributions well within domain adaptation and causal representation learning. It cites relevant works on domain adaptation, causal disentanglement, and latent variable identifiability​.", "essential_references_not_discussed": "As far as I know, the authors have included relevant and recent references related to the subject matter.", "other_strengths_and_weaknesses": "Strengths\n- Strong theoretical foundation for identifiability.\n- Clear motivation for partitioning the Markov blanket.\n- Well-structured experimental validation.\n\nWeaknesses\n- Some theoretical assumptions may be restrictive.\n- Lack of discussion on computational efficiency.\n- No discussion on potential failure cases or limitations.", "other_comments_or_suggestions": "- Provide additional baselines from causal representation learning.\n- Discuss the computational complexity of the approach.\n- Clarify the generalizability of the method beyond the considered datasets.\n- Page 7, line 372 first column. I think the authors would refer to Figure 2 instead of Figure 1.", "questions_for_authors": "1. How does the approach handle highly imbalanced domain shifts?\n2. Would the method still be effective with fewer source domains?\n3. Are there any practical limitations regarding computational resources for training?\n4. Could the theoretical results be extended to other types of unsupervised domain adaptation problems?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIqlswUaeO", "reviewer": "Reviewer_d2cE", "summary": "This paper addresses the issue of multi-source domain adaptation with a focus on identifiability. It introduces a causal framework that avoids restrictive assumptions such as independent latent variables or invariant label distributions. The authors theoretically establish identifiability and validate the effectiveness of their method through experiments on datasets like Office-Home and PACS.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive comments, helpful feedback, and time devoted. Please see below for our response.\n\n**Q1:** \"the experiments are limited to datasets with covariate shifts\" and \"experiments involving other types of distribution shifts\"\n\n**A1:** Thanks for your comment. Notably, several existing works [1, 2, 3] on label shifts have also used Office-Home and PACS, highlighting the presence of label shifts in these datasets. A possible reason is that, to the best of our knowledge, there is no benchmark dataset specifically designed for label shift. If you are aware of any such datasets, we would greatly appreciate your suggestions and would be happy to incorporate them into our analysis.\n\nTo validate the presence of label shifts, we analyzed the label distributions of Office-Home and PACS. The visualizations are provided in Figures 1, 2, and 3 in https://anonymous.4open.science/r/icml-rebuttal-gama/rebuttal.pdf. Specifically, Figures 1 and 2 illustrate the label distributions across different domains for Office-Home and PACS, respectively, indicating notable variations in label probabilities across domains. To further quantify this, we computed the Jensen–Shannon divergence between label distributions for different domain pairs (Figure 3), confirming clear label shifts—especially in the Art sub-task of Office-Home and the Sketch sub-task of PACS. Thus, beyond covariate shift as you pointed out, both datasets also exhibit label shifts.\n\nTo further investigate our method’s robustness to severe label shift, we created a more extreme setting for Clipart sub-task (from Office-Home dataset ) by sampling data points from different labels according to a pre-defined label distribution; see Figure 4 in https://anonymous.4open.science/r/icml-rebuttal-gama/rebuttal.pdf, where the label distribution of Clipart domain differs substantially from other domains. The corresponding Jensen–Shannon divergence (Figure 5) verifies this and indicates a much more severe label shift for Clipart. For this sub-task, we compared our method against iMSDA, one of the strongest baselines in our experiments. Our method achieves an accuracy of $57.7$, outperforming iMSDA that achieves $57.1$, demonstrating the effectiveness of our approach under severe label shift.\n\nWe will incorporate this discussion and additional experiment into the revision. Hope this addresses your concern.\n\n**Q2:** \"Including the latest baselines would enhance the rigor of the analysis\"\n\n**A2:** In light of your suggestion, we will include 5 additional recent baselines (after 2022) in the revision, i.e., CASR [4], TFFN [5], SSD [6], GeNRT [7], and iLCC-LCS [8]. The results are provided in the tables below, which indicate that our method achieves superior performance.\n\n- Office-Home dataset:\n\n| Method      | Ar   | Cl   | Pr   | Rw   | Avg  |\n|-------------|------|------|------|------|------|\n| CASR [4]    | 72.2 | 61.1 | 82.8 | 82.8 | 74.7 |\n| TFFN [5]    | 72.2 | 62.9 | 81.7 | 83.5 | 75.1 |\n| SSD [6]     | 72.5 | **64.5** | 81.2 | 83.2 | 75.4 |\n| **GAMA (Ours)** | **76.6** | 62.6 | **84.9** | **84.9** | **77.3** |\n\n- PACS dataset:\n\n| Method      | P    | A    | C    | S   | Avg  |\n|-------------|------|------|------|------|------|\n| GeNRT [7]    | 98.5 | 93.6 | 91.4 | 85.7 | 92.3 |\n| iLCC-LCS [8]    | 95.9 | 86.4 | 81.1 | 86.0 | 87.4 |\n| **GAMA (Ours)** | **98.8** | **93.7** | **92.8** | **89.3** | **93.7** |\n\n**Q3:** \"Why did the authors choose a multi-VAE structure for implementation? Could the author provide more insights into their intuition behind this choice?\"\n\n**A3:** Thanks for your question, which helps clarify the motivations behind our implementation. First, VAE provides a convenient way to model the distribution of latent variables. Second, compared to other generative models, VAEs make it easier to incorporate prior structural information (e.g., parent-child relationships) into our method. Using multiple VAEs further facilitates the integration of such structural priors. We will clarify this in the revision.\n\n---\n**References:**\n\n[1] Le et al., On Label Shift in Domain Adaptation via Wasserstein Distance. arXiv, 2022.\n\n[2] Jang et al., Distribution Shift-Aware Prediction Refinement for Test-Time Adaptation. arXiv, 2024\n\n[3] Liu et al., Domain Generalization under Conditional and Label Shifts via Variational Bayesian Inference. In IJCAI, 2021.\n\n[4] Wang et al., Classaware sample reweighting optimal transport for multi-source domain adaptation. Neurocomputing, 2023.\n\n[5] Li et al., Transferable feature filtration network for multi-source domain adaptation. Knowledge-Based Systems, 2023.\n\n[6] Li et al., Multidomain adaptation with sample and source distillation. IEEE Transactions on Cybernetics, 2023.\n\n[7] Deng et al., Generative model based noise robust training for unsupervised domain adaptation. arXiv, 2023.\n\n[8] Liu et al., Identifiable latent causal content for domain adaptation under latent covariate shift. arXiv, 2024.", "claims_and_evidence": "All the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are suitable for the problem.", "theoretical_claims": "Due to limited time, I couldn't check every detail of the derivation. However, most of the theorems seem intuitive and appear to be correct.", "experimental_designs_or_analyses": "1. The primary concern with this work is that while it claims to \"consider the most general scenario\" and suggests adaptability to different data distribution shifts, the experiments are limited to datasets with covariate shifts. It would be more convincing if the authors also conducted experiments involving other types of distribution shifts, such as label shift or conditional shift.\n2. The most recent domain adaptation (DA) baselines included in the study are from 2022. Including the latest baselines would enhance the rigor of the analysis.", "relation_to_broader_scientific_literature": "The works most closely related to this are [1] and [2]. However, [1] adopts more stringent assumptions about the independent latent variables, whereas [2] does not ensure identifiability.\n\n[1] Kong, Lingjing, et al. \"Partial disentanglement for domain adaptation.\" International conference on machine learning. PMLR, 2022.\n[2] Zhang, Kun, et al. \"Domain adaptation as a problem of inference on graphical models.\" Advances in neural information processing systems 33 (2020): 4965-4976.", "essential_references_not_discussed": "As far as I known, all the essential references are discussed in the paper.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The novel multi-source adaptation framework introduced in the paper guarantees identifiability under mild restrictions.\n2. Effectiveness of the theorem and its implementation are verified through experiments conducted on datasets such as Office-Home and PACS.\n3. The paper is well-written and easy to follow.\n\n**Other Weaknesses:**\n\nNone. Most potential weaknesses are addressed in the \"Experimental Designs\" section.", "other_comments_or_suggestions": "None", "questions_for_authors": "The theoretical framework proposed by the author suggests there are multiple ways to implement it in practice. Why did the authors choose a multi-VAE structure for implementation? Could the author provide more insights into their intuition behind this choice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel contribution for multi-source Domain Adaptation based on a identifiability framework and the use of VAE.


During evaluation, the novelty of the paper and the strong theoretical background  of the contribution have been noted, however some limitations in the experiments, the analysis and the discussion were also raised.  
During rebuttal, authors have provided different answers to the reviewers. 
After the rebuttal, the three reviewers maintained a positive evaluation (2 weak accepts and 1 accept) and agreed on the recommendation to accept the paper.
Decision: Accept (poster)