Latent Covariate Shift: Unlocking Partial Identifiability for Multi-Source Domain Adaptation
Authors: Yuhang Liu
Zhen Zhang
Dong Gong
Mingming Gong
Biwei Huang
Anton van den Hengel
Kun Zhang
Javen Qinfeng Shi
Venue: Accepted by TMLR ()

Abstract: Multi-source domain adaptation (MSDA) addresses the challenge of learning a label prediction function for an unlabeled target domain by leveraging both the labeled data from multiple source domains and the unlabeled data from the target domain. Conventional MSDA approaches often rely on covariate shift or conditional shift paradigms, which assume a consistent label distribution across domains. However, this assumption proves limiting in practical scenarios where label distributions do vary across domains, diminishing its applicability in real-world settings. For example, animals from different regions exhibit diverse characteristics due to varying diets and genetics.

Motivated by this, we propose a novel paradigm called latent covariate shift (LCS), which introduces significantly greater variability and adaptability across domains. Notably, it provides a theoretical assurance for recovering the latent cause of the label variable, which we refer to as the latent content variable. Within this new paradigm, we present an intricate causal generative model by introducing latent noises across domains, along with a latent content variable and a latent style variable to achieve more nuanced rendering of observational data. We demonstrate that the latent content variable can be identified up to block identifiability due to its versatile yet distinct causal structure. We anchor our theoretical insights into a novel MSDA method, which learns the label distribution conditioned on the identifiable latent content variable, thereby accommodating more substantial distribution shifts. The proposed approach showcases exceptional performance and efficacy on both simulated and real-world datasets.

Published in Transactions on Machine Learning Research (04/2025)
Latent Covariate Shift: Unlocking Partial Identifiability for Multi-Source Domain Adaptation
Yuhang Liu
yuhong. linnn @adelaide edu..on
Australian Institute for Machine Learning The University of Adelaide
Zhen Zhang
zhen. Sanr.ahhnnOO@@daaaadeeee edu.
Australian Institute for Machine Learning The University of Adelaide
Dong Gong
dong. Mong.gonggee www.ggnggguuuu edu.
School of Computer Science and Engineering The University of New South Wales
Mingming Gong
www.hinggggggg@inmmill.... edw.oo
School of Mathematics and Statistics The University of Melbourne
Biwci Huang
bih:07 edu
Halicioguu Data Science Institute University of California San Diego
Anton van den Hengel
antor.vwaneenienng@@ddddddd.d... edu.
Australian Institute for Machine Learning University of Adelaide
Kun Zhang
kuns @cmu. edu
Department Philosophy
Carnegie Mellon University
Javen Qinfeng Shi
javen.shi@uadelaidecc edou.
Australian Institude for Machine Learning The University of Adelaide
Reviewed on OpenReview: https: /ttps://openreeee opwneeveee........ net/ forum? 3799
Abstract
Multi-source domain adaptation (MSDA) addresses the challenge of learning label pre- diction function for unlabeled target domain by leveraging both the labeled data from multiple source domains and the unlabeled data from the larget domain. Conventional MSDA approaches often rely on covariate shift or conditional shift paradigms, which assume a consistent label distribution across domains. However, this assumption proves limiting in practical scenarios where label distributions do vary across domains, diminishing its appli- cability real-world settings. For example, animals from different regions exhibit diverse characteristies due to varying diets and genetics.
Motivated by this, we propose a novel paradigm called latent covariate shift (LCS), which introduces significantly greater variability and adaptability across domains. Notably, pro- vides thcoretical assurance for recovering the latent cause of the label variable, which we refer the latent content variable. Within this new paradigm, we present an intricate causal generative model by introducing latent noises across domains, along with latent

---

Published in Transactions on Machine Learning Research (04/2025)
content variable and latent style variable to achieve more muanced rendering of observa- lional data. We demonstrate that the latent content variable can be identified up to block identifiability due to its versatile yet distinct causal structure. We anchor our theoretical insights into a novel MSDA method, which Iearns the label distribution conditioned on the ifeblifiable latent content variable, thereby accommodating more substantial distrriuttinn shifts. The proposed approach showcases exceptional performance and efficacy on both simulated and real-world datasets.
The code available at: httpp://sttteego....../////aa/ap
1 Introduction
Multi-source domain adaptation (MSDA) aims to utilize labeled data from multiple source domains and unlabeled data from the targel domain, learn model to predict. well in the larget domain. Formally, denoting the input as (e.g., an image), y labels in both source and target domains, and as the domain index, during MSDA training, we have labeled source domain input-output pairs, (x5, .yy)) drawn from source domain distributions Pu=ui (x,y), Pu=u.,, yx Note that the distribution (x, may vary across domains. Additionally, we observe some unlabeled target domain input data, xJ, sampled from the target domain distribution Pur Pur(x,y). yx.
The success of MSDA hinges on two crucial elements: variability in the distribution Pu(x, y), determining the extent to which it may differ across domains, and the imperative of invariability in certain portion of the same distribution to ensure effective adaptation to the target domain. Neglecting to robustly capture this invariability, often necessitating a theoretical guarantee, hampers adaptability and performance in the target domain. Our approach comprehensively addresses both these elements, which will be elaborated upon in subsequent sections.
u
X
Z
Ze
(a) Covariate Shift
(b) Conditional Shift:
(c) Latent Covariate Shift
Figure The illustration of three different paradigms for MSDA. Covariate Shift: Px(x) changes across domains, while Pa (x) is invariant across domains. Conditional Shift: Paly) is invariant, while pa (xly) changes across domains. Latent Covariate Shift: (a.) changes across domains while puly (2)) invariant.
MSDA can be broadly classified into primary strands: Covariate Shift (Huang
2006 Bickel al,
2007 Sugiyama al., 2007 Wen et al.) 2014 and Conditional Shift (Zhang et al. 2013 2015 Schölkopf
et al. 2012 Stojanov et al. 2021 Peng et al. 2019). In the early stages of MSDA research, MSDA methods focus on the first research strand Covariate Shift as depicted by Figure [(a) It assumes that (x) changes across domains, while the conditional distribution Pu (y|x) remains invariant across domains. However, this assumption does not always hold in practical applications, such as image classification. For example, the assumption of invariant Puly xx) implics that Pu(y) should change as Pu(x) changes. Yet, we can easily manipulate style information (e.g., hue, view) in images alter Pu while keeping Pu(y) unchanged, which clearly contradicts the assumption. In contrast, most recent works delve into the Conditional Shift as depicted by Figure [bb) Tt assumes that the conditional Pa (x changes while Pu (y) remains invariant across domains
(Zhang et al. 2013 2015 Schölkopf et al. 2012 Stojanov ct al. 2021
Peng et al.
2019
Conscquently,
it has spurred a popular class of methods focused on learning invariant representation ionss across domains Lo
target the latent content variable ze in Figure 1(b) (Ganin et al. 2016 Zhao et al.]
2018 Saito et al. 2018
Mancini et al.) 2018 Yang et al., 2020 Wang et al. 20200 Li al. 2021 Kong et al.
2022). However,
the label distribution Py(y) can undergo changes across domains in many real-world scenarios Taccet des
Too simplify our notation without introducing unnecessary complexity, we emplay notation Pu=u., (x.y) y) denote p(x, y/u Up), and express Um as U Um with the aim clarity.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Combes et al. 2020: Lipton et al. 2018 Zhang et al. 2013), and enforcing invariant representations can degenerate the performance (Zhao et al.. 2019).
In many real-world applications, the label distribution Py(y) exhibits variation across different domains. For instance, diverse geographical locations entail distinct species sets and/or distributions. This characteristic is well-illstrratd by a recent, meticulously annotated dataset focused on studying visual generalization across various locations (Beery et al., 2018). Their findings illuminate that the species distribution presents a long-tailed pattern at each location, with each locale featuring distinct and unique distribution 0 Label
distribution shifts can also be corroborated through analysis the WILDS benchmark dataset, which
investigates shifts in distributions within untamed and unregulated environments (Koh ct al. 2021 This
study uncovers pronounced disparities in label distributions between Africa and other regions These distinctions encompass a notable decrease in recreational facilities and a marked rise in single-unit residential properties.
To enhance both critical elements of MSDA, namely variability and invariability, we introduce novel paradigm termed Latent Covariate Shift (LCS), as depicted in Figure [[c] Unlike previous paradigms, LCS introduces latent content variable Ze the common cause of and y. The distinction between LCS and previous paradigms is detailed in Section 2 In essence, LCS allows for the flexibility for Pu(ze), Pu(x), Py()) and (x(2x to vary across domains (greater variability). Simultaneously, its inherent causal structure guarantees that Pu (y(2z) remains invariant irrespective of the domain (invariability with assurance). This allowance for distributional shifts imparts versatility and applicability to a wide array real-world problems, while the stability of Pu(ylze) stands as the pivotal factor contributing to exceptional performance across domains.
Within this new paradigm, present an intricate latent causal generative model, by introducing the latent style variable z, in conjunction with Ze, as illustrated in Figure 2(a) We delve into an extensive analysis of the identifiability within our proposed causal model, affirming that the latent content variable ze can be established up to block identifiability] through rigorous theoretical examination. Since Zg and the label become independent given 2e, it is no need to recover zg. The identifiability on ze provides a solid foundation for algorithmic designs with robust theoretical assurances. Subsequently, we translate these findings into a novel method that learns invariant conditional distribution Pu(y/ze), known as independent causal mech- anism, for MSDA. Leveraging the guaranteed identifiability of 2e, our proposed method ensures principled generalization to the target domain. Empirical evaluation on both synthetic and real-world data showcases the effectiveness of our approach, outperforming state-of-the-art methods.
2 Related Work
Learning invariant representations. Due to the limitations of covariate shift, particularly in the context of image data, most current research on domain adaptation primarily revolves around addressing conditional shift. This approach focuses on learning invariant representations across domains, a concept explored in works such as Ganin et al. (2016); Zhao et al. (2018): [Saito et al. (2018): Mancini et al. (2018); Vang et al. (2020g: Wang et al. (2020): Li ct al. (2021): WWan et al. (202bb): : Zhhoo et al. (2021): Kong et al. (2022); Wen al. (2024). These invariant representations are typically obtained by applying appropriate linear nonlinear transformations to the input data. The central challenge in these methods lies in enforcing the invariance of the learned representations. Various techniques are employed to achieve this, such as maximum classifier discrepancy (Saito el al., 2018), domain discriminator for ad adversarial training (Ganin el al.] 2016 Zhao et al. 2018 2021), moment matching (Peng et al., 2019). and relation alignment loss (Wang et al. 2020). However, all these methods assume label distribution invariance across domains. Consequently, when label distributions vary across domains, these methods may perform well only in the overlapping regions of label distributions across different domains, encountering challenges in areas where distributions do not overlap. To overcome this, recent progress focuses on lcarning invariant reprosentations conditional the
FFor comprchensive depiction of these distributions, please refer to Figure 4 in Beery al. APPENDIX offers an analogous representation, as shown in Figure 7 for completcness.
2018
Additionally, the
For thorough comparison of distributions, refer Figures 24 and 27 in Koh et 2021 4There exists invertible function between the recovered 2, and the true one Von Kügelgen gen al. 2021
3

---

Published in Transactions on Machine Learning Research (04/2025)
label across domains (Gong al. 2016 Ghifary et al. 2016 Tachet des Combes et al. 2020 One of the challenges in these methods is that the labels in the target domain are unavailable. Moreover, these methods do not guarantee that the learned representations align consistently with the true relevant information. A recent work Kong et al. (2022) introduces high-level invariant variables, which differ from traditional invariant variables by identilying them through the transfer of domain index-based variant variables to invariant ones. However, this approach remains limited to the context of invariant label distributions across domains.
Learning invariant conditional distribution p.,y/..)) The investigation of learning invariant condi- tional distributions, specifically Pu(y|ze). for domain adaptation has seen limited attention compared to the extensive emphasis on learning invariant representations (Kull and Flach 2014 Bouvier et al. 2019). What sets our proposed method apart from these two works is its causal approach, providing identifiabilit, for the true latent content Ze, a significant challenge in latent causal models (Liu et al. 2022 2024ch. This serves as thcoretical guarantee for capturing invariability, addressing the second key element of MSDA. In other words, it ensures that the Icarned Pa (y/ze) in our work can generalize to the target domain in a principled manner. In addition, cerlain studies explore the identification of py(y(e)) by employing a proxy variable, as demonstrated by (Alabdulmohsin et al. 2023). The challenge of these studies lies in devising proxy variable. In contrast, although the work do not need such proxy variable, it is worth noting that our work may necessitate stronger assumptions for identifying latent Ze, compared with proxy based methods. Additionally, our Latent Causal Structure (T.CS) allows for more flexibility in accommodating variability, addressing the first key element of MSDA. Besides, in the context of out-of-distribution generalization, some recent works have explored the learning of invariant conditional distributions Pu (y(z.) (Arjovsky et al. 2019: Sun et al. 2021 Liu et al. 2021: Lu et al. 2021). For example, Arjovsky et al. (2019) impose learning the op- timal invariant predictor across domains, while the proposed method directly explores conditional invariance by the proposed latent causal model. Moreover, some recent works design specific causal models tailored to different application scenarios from a causal perspective. For example, Liu et al. (2021) mainly focus on a single source domain, while the proposed method considers multiple source domains. The work by [Sun et al. (202I) explores the scenarios where a confounder to model the causal relationship between latent content variables and style variables, while the proposed method considers the scenarios which latent style variable is caused by content variable. The work in (Lu et al., 2021) focuses on the setting where label variable is treated variable causing the other latent variables, while in our sccnarios label variable has child nodes. A very recent work by (Li et al., 2023 also aims to learn such an invariant conditional distribution. However, unlike ours, their approach assumes that the latent space can be split into four distinct subspaces. This specific and detailed partitioning introduces additional assumptions, which may impose unnecessary constraints and limit applicability in real-world scenarios, potentially leading to suboptimal results. We provide experimental comparisons on real datasets to substantiate this claim.
efficient
Causality for Domain Generalization A strong connection between causality and generalization has been established in the literature (Peters et al. 2016), primarily focusing on leveraging invariant mechanisms for generalization, often through causal discovery (Zhang et al. 2022 2025). Building upon this insight, current rescarch has leveraged causality to introduce novel methods across various applications, including domain generalization (Mahajan el al. 2021 Christiansen et al. 2021 Wang el al. 2022a), text classification (Veitch) ett al. 2021), and Out-of-Distributtin gencralization closely related to our problem sctting. However, involves scenarios where the input data in the target domain cannot be directly observed. The lack of access to the input data makes obtaining identifiabiliyy results challenging, and is left for future work.
Generalization (Ahuja et al.) 2021). Among these applications, domain
3 The Proposed Latent Causal Model for LCS
LCS is a new paradigm in MSDA, enriching the field with elevated variability and versatility. Within this innovative framework, we're presented with an opportunity to delve into more intricate models. This section is dedicated to presenting a refined causal model, failored to this paradigm.
LCS introduces a novel framework to MSDA, providing flexible framework that incorporates a wide range of latent causal generative models. These models can be customized for different application scenarios, allowing the integration of various causal structures and latent variables according to the specific challenges
4

---

Published in Transactions on Machine Learning Research (04/2025)
P2
4
(a) The proposed causal model
(b) An equivalent graph structure
Figure 2: (a) The proposed latent causal model, which splits latent noise variables n into two disjoint parts, ne and n, (b) An equivalent graph structure, which can generate the same observed data x as obtained by (a), resulting a non-identifiabliiy result.
and characteristics of each domain. This adaptability enables LCS to effectively capture underlying causal relationships and manage complex distribution shifts across multiple sources. For instance, in a specific application scenario, we incorporate a latent style variable that works alongside the latent content variable. This enables more nuanced modeling of domain-specifie factors, such as style variations in image data, while preserving core content information across domains. By jointly modeling both content and style, the framework is better equipped to capture complex relationships between sources and enhance generalization performance in domain adaptation tasks.
Fundamentally, the latent content exerts direct influence over the available styles. To account for this, we introduce a latent style variable, seamlessly integrated as a direct descendant of the latent content. variable, as depicted in Figure [[a] Concurrently, the observed domain variable u serves as an indicator of the specific domain from which the data is sourced. This domain variable gives rise two distinct groups of latent noise variables: latent content noise, denoted as ne, and latent style noise, denoted ng. Analogous exogenous variables in causal systems, these elements play pivotal roles in shaping both the latent content variable z, and the latent style variable 2,.
This model stands out in terms of versatility and applicability across a wide range of real-world scenarios, surpassing the capabilities of models in previous paradigms including covariate shift and conditional shift. This because accommodates variations in Pu(ze), Pu(x) and Pu(x|ze) across domains. Moreover, it provides theoretically established guarantee of invariability for Pu(y (2.) independent of the domain (see Sections and 5). This pivotal property facilitates the model's ability to generalize predictions across diverse array of domains. To parameterize it, we make the assumption that n follows an exponential family ion given u, and we describe the generation of z and x as follows:
E
2
(n/u) =
exp
(Tis (1)))))a
(1)
(u)
i=1
j=1
z. ge (n.), 2, 8.2 go1 (go1 (z.) +ns),
(2)
x = f(zc, Z,) +
(3)
In Eq. 1 (u) represents the normalizing constant, (m,) stands as the sufficient statistic for n,, and denotes the number of latent noises. The natural parameter (u) dependent on the domain variable u. To establish a coherent connoction between these latent noise variables n and the latent variables z, we employ post-nonlinear models, defined in Eq. 2 where ge and ga2 are invertible functions. This concept of post-nonlinear models (Zhang and Hyvarinen 2012 represents generalized form of additive noise models (Hoyer et al. 2008), which find widespread use across various domains. Furthermore, our proposed model incorporates two fundamental causal assumptions that serve to underscore its novelty and distinctiveness within the field of latent. causal modeling.
2, causes y: Prior research has often considered the causal relationship between x and y as y + GGog et al. 2016 Stojanov et al., 2019 Li et al. (2018). In contrast, our approach employs Ze + y. Note that these two are not contradictory, they pertain to different interpretations of the labels y representing
5

---

Published in Transactions on Machine Learning Research (04/2025)
distinct physical meanings. To clarify this distinction, let us denote the label in the first case as (i.e.; ÿ x) to distinguish it from y in the second case (i.e., 3e y). In the first case, consider the generative process for images: a label, 9, is initially sampled, followed by the determination of content information based on this label, and finally, the generation of an image. This sequence aligns with reasonable assumptions in real-world applications. In our proposed latent causal model, we introduce ne to play a role similar to 9 and establish a causal connection with the content variable Ze- In the second case, Ze -> y represents the process where experts extract content information from provided images and subsequently assign appropriate labels based on their domain knowledge. This assumption has been adopted by recent works (Mahajan et al. 2021; Liu et al. 2021 Sun et al. 2021). Notably, both of these distinct labels, 9 and y, have been concurrently considered in Mahajan et al. (2021). In summary, these two causal models, & + x and &e + y, capture different aspects of the generative process and are not inherently contradictory, reflecting varying perspectives on the relationship between labels and data generation.
Ze causes z₁: As discussed carlier, the underlying content directly molds the styles. Therefore, we adopt the causal relationship where 2, serves as the cause of Zg. In real applications, for example, different animals (e-g.: ze) often choose different environments (e.g., z.) where the basic needs of the animals to survive are met. This can be interpreted as the essence of the object, Ze, being the primary factor from which latent style variable, Zg, emerges to ultimately render the observation x. This is aligned with previous works in domain adaptation field (Gong et al. 2016 Stojanov et al. 2019 Mahajan et al. 2021), ), and with advancements in self-supervised learning (Von Kügelgen et al. 2021
Daunhawer et al. 2023 Cai et al, 2024).
Comparison with Previous Works The proposed model introduces an intermediate latent variable Ze: which enhances interpretability and disentanglement in latent space compared to the previous model ZZhng et al.] (013)) that employs + y + x, neglecting such high-level latent variables. By structuring the u - ze - y and ze) + x, our approach better captures indirect effects compared to u + y + x, handles distribution shifts more robustly, and supports clearer intervention modeling. For example, for the latter model, once we determine y, then x is also determined. However, consider an animal dataset: even if we fix the animal label y: the observed features x may still change due to variations in background caused by environmental shifts. This highlights a limitation of the direct causal structure u u-y-t + x, as it fails to account for latent factors such as habitat conditions, lighting, or seasonal effects that influence x. Our model, by introducing 2 in Figure 2, specific case of the proposed model, allow for all these variations, leading to better robustness under distribution shifts. In addition, compared to previous work Tachet des Combes et al. (2020), which also considers label distribution shift, we formulate the problem from the perspective of a latent causal generative model and provide a theoretical guarantee for recovering the latent variables Ze- In contrast, the previous work assumes the existence of invariant representations given y, which is challenging to learn due to the unknown nature of y in the target domain.
causal pathways
4 Identifiability Analysis
In this section, we provide an identifiability analysis for the proposed latent causal model in section 3 We commence by illustrating that attaining complete identifiability for the proposed causal model proves unattainable without the imposition of stronger assumptions. This assertion is substantiated by the con- struction of an alternative solution that deviates from the true latent causal variables, yet yields the same observational data. Following this, we move forward to affirm that achieving partial identifiability of the latent content variables, denoted as Ze, up to block identifiability is within reach. This level of identifiability alrcady provides a solid foundation for guiding algorithm designs with robust thcoretical guaranteos, while maintaining the necessary flexibility for adaptation. Indeed, we can see that Z, serves as a common cause for both y and 2g, illustrating a typical scenario of spurious correlation between y and Z,. Once ze is successfully recovered, it alone contains ample information to make accurate predictions about the label. Given Ze, the label and the latent style become independent. Thus, it is unnecessary and potentially counterproductive recover the latent style variable for label prediction, when the latent. content has already been ret rieved.
6

---

Published in Transactions on Machine Learning Research (04/2025)
4.1 Complete Identifiability: The Non-identifiability Result
Given the proposed causal model, one of the fundamental problems is whether can uniquely recover the latent variables, i.e, identifiability. We show that achieving complete identifiability of the proposed latent causal model remains a challenging endcavor, as follows:
Proposition 4.1 Suppose that the latent causal variahles a and the observed variable follow the latent causal models defined in Eq. [ given observational data distribution ply, x x()), there erists an alternative solution, which can yield exactly the same observational distribution, resulting in non-identifabbityy without further ussumplions.
Proof sketch The proof of proposition 1.1 can be done by proving that we can always construct another alternative solution to generate the same observation, leading to the non-identiffabiliyy result. The alternative solution can be constructed by removing the edge from Ze to Zg as depicted by Figure 2(b) i.e., 2 n. (ne), 2 = and the mixing mapping from z' to is a composition function, e.g., fo f where z.)].
f((z)) = -
Intuition The non-identifiability result above because we can not determine which path is the correct path corresponding to the net effect of n. on x; i.e., both n. + Ze - 2, + x in Figure 2(a) and ne + Ze + x in Figure 2(b) can generate the same observational data x. This problem often appears in latent causal discovery and seriously hinders the identifiability of latent causal models.
4.2 Partial Identifiability: Identifying be
While the above result in proposition 4.1 shows that achieving complete identifiability may pose challenges, for the specific context of domain adaptation, our primary interest lies in the identifiability of Ze: rather than the latent style variable Zg. This focus is justified by the fact that the label is solely caused by Ze. In fact, we have established the following partial identifiability result:
Proposition 4.2 Suppose latent causal variables z and the observed variable x follow the generative models defined in Eq. 1 Eq. Assume the following holds:
(1) The set {x E (x)x=(x)= 0} has measure zero (i.e., has al the most counlable number of elements), where Pe is the characteristic function of the density Pe-
(i) The function f in Eq. bijective.
(iii) There exist 28 + 1 distinct points un u1 Usz such that the matric
== (1 un) - (n =o) uo)..., (, u2e) - -y(u = =un))
(4)
of size 28 x 28 is invertible where 17(1= [m:,(u))k,,
then the recovered latent content variables že, which are learned by matching the true marginal data distribu- tion p(x u) and by using the dependence between n and y conditional on are related to the true latent causal variables ze by the following relationship: z. = h(2,), where h denotes an invertible mapping.
Assumptions (ii- are motivated by the nonlinear ICA literature (Khemakhem et al. 2020 which is to provide guarantee that can recover latent noise variables up to permutation and scaling transforma- tion. The main Assumption (iii) essentially requires sufficient changes in latent noise variables to facilitate their identification, which has also been introduced in recent advances in causal representation learning Liu et al. (2024b a). Furthermore, we conduct model assumptions, as defined in Eqs. I Essentially, Eq. L de- rived from Sorrenson et al. (2020), posits that each n, is sampled from the wwo-parameter exponential family. We adopt this assumption consideration of woal-world applications where the dimension of n, unknown. By assuming two-parameter exponential family members, it has been demonstrated that informative latent
7

---

Published in Transactions on Machine Learning Research (04/2025)
noise variables n, can be automatically separated from noise by estimating model (for more details, refer to Sorrenson et al. (2020)). However, it is important to acknowledge that in real applications, the distribu- tion of n could be arbitrary. In this context, assumption Eq. [ may only serve as an approximation for the true distribution of n. Nevertheloss, in terms of the performance of the proposed method on real datascts such as PACS and TerraIncognita, such an approximation may be deemed acceptable to some exlent. To establish a coherent connection between these latent noise variables n and the latent variables z, we assume post-nonlinear models, as defined in Eq. 2 We posit that post-nonlinear models could be further relaxed, as long as the assumed models are invertible from to Here, we choose to employ post-nonlinear models, considering their ease of parameterization and understanding.
Remark 4.3 With access to label y in source domains, and the identified ze in those domains, it becomes possible to accurately estimate the parameters of the conditional distribution p(y|z.). This allows for robust modeling of the relationship between the latent content variables Ze and the corresponding labels y in the source domains. Then since ze in target domain is also identified, and leveraging the invariance of ply|ze), the learned conditional distribution p(y|ze) theoretically be generalized to the target domain. This generalization is grounded in the notion that the causal relationship between Ze and y remains consistent across domains.
Proof sketch The proof of Proposition 4.2 can be outlined as follows: Given that the mapping from n to z is invertible, and in consideration of assumptions (i)-(ii)). we can leverage results from nonlinear ICA (Hyvarinen et al., 2019 Khemakhem et al. 2020 Sorrenson et al. 2020). This implies that n can be identified up to permutation and scaling, i.e., n Pá + c, where ñ represents the recovered latent noise variables. Furthermore, the graph structure depicted in Figure 2(a] implies that yd is dependent on no but independent of n,, given u. This insight enables us to climinate the permutation indeterminacy, thus allowing us to identily n.- Ultimately, since the mapping from D. to 2 is invertible as defined in Eq. 2 we can conclude the proof.
We note that all three assumptions align with standard practices in the nonlinear ICA literature [Hyvrinnen et al. 2019 Khemakhem et al., 2020). This alignment is possible because nonlinear ICA and latent causal models naturally converge through the independence of components in the nonlinear ICA domain and the independence among exogenous variables in causal systems. However, there exists a crucial distinction: while nonlinear ICA seeks to identify independent latent variables, our objective in this context is to pinpoint the latent content variable within causal system that allows for causal relationships among latent variables. This inherently renders the task more challenging. Fortunately, this discrepancy can be reconciled through two pivotal conditions: 1) the mapping from to z is made invertible by constraining the function class, as defined in Eq. 2 2) The conditional independence between n and ys, given u, as depicted in the graph structure shown in Figure 2(a) This condition effectively eliminates the permutation indeterminacy typically encountered nonlinear TCA. Furthermore, Proposition 4.2 establishes the existence of an invertible transformation between the recovered a. and the true latent content variable ze. Importantly, this invertible transformation has no bearing on domain adaplation lasks, as it indicates that 2, encapsulates all and only the information pertaining to Ze.
Intuition
5 Learning Independent Causal Mechanism Pu Py(()))) for MSDA
The identifiability of ze provides a solid foundation for cnsuring that the independent causal mechanism Pu(y|ze)) can be effectively learned across different domains. This, in turn, supports robust generalization to the target domain. Moreover, given the invertible mapping between ne and Ze, shown in Eq. 2 the task of learning Pa(y/z.) can be reformulated as the task of learning pu(y)n.). This reformulation remains consistent and invariant across domains, as demonstrated in Figure ((a)
As elucidated in Proposition 4.2 the learned latent content variables 2, are obtained by aligning with the true marginal data distribution p(x u). Consoquently, it logically follows that the acquired latent noise variables f should also be obtained through a matching of the marginal distribution. To fulfill this objective, our proposed method harnesses the framework of Variational Autoencoder (Kigmma and Welling 2013) to learn
8

---

Published in Transactions on Machine Learning Research (04/2025)
ELBO
Classifier
Prior
P2(n)
Transfer
Clasofier
9,(m,
Encoder
g,(n,
Decoder
Figure 3: The proposed iLCC-LCS to learn the invariant p(y)n,) for multiple source domain adaptation. C denotes concatenation, and S denotes sampling from the posterior distributions.
the recovered latent noise. Specifically, employ Gaussian prior distribution for n. and n, as follows: (n) = (a,n.)Pa(nn)) = (pn, (u), En, (u)) (Pn, (u), Eu) (u)), (u))):
(5)
where fu and E denote the mean and variance, respectively. Both μ and E depend on the domain variable u and can be implemented with multi-layer perceptrons. Since providing the true posterior in general challenging (Liu et al. 2018 2019), we turn to the following variational posterior to approximate it:
qu (u(n/x) qu (n) (N'',,
(u,
(u, xu, (u,x)),
(6)
where ul and V denote the mean and variance of the posterior, respectively. Combining the variational posterior with the Gaussian prior, we can derive the following evidence lower bound:
LELBO (m) (logpr())) 3Dxc (I))))))))l
(7)
where DEc denotes the Kullback-Leibb divergence. We here empirically use hyperparameter 3, motivated by Higgins et al. (2017); Kim and Mnih (2018): ; Chen et al. (2018), to enhance the independence among n4, considering common challenge encountered in real-world applications where the availability of source domains is often limited. By maximizing the Evidence Lower Bound (ELBO) as expressed in Eq. 7 we can effectively recover n; up to scaling and permutation. To address the permutation indeterminacy, as shown in proposition 4.2 we can evaluate the dependence between y and nę to identify which n, correspond to n In the implementation, we use the variational low bounder of mutual information as proposed in Alemi et al. 2016)) quantify the dependence as follows:
M By (logr(y~(nn) In
This loss function serves maximize the mutual information between n. and y in source domains. Notably, this maximization also signifies an amplification of the information flow within the causal relationship from o for y This alignment between information flow and mutual information arises from the independence among n,, conditioned on u, which is a structural characteristic inherent in the graph representation of our proposed causal model. To promote information flow in the target domain, we can also maximize the mutual information between TT (representing the estimated label the target domain) and n.. This is achieved by minimizing the following conditional entropy term:
LENT -Eqg(n (x)
(9)
where S denotes the estimated label the target domain. It obtained using the encoder component of the VAE framework, which is trainod with the LELBO loss to derive n.. This latent representation then passed into classifier that. is trained on nf and y$ from the source domain. It is interesting note that
this regularization approach has been empirically utilized in previous works to make label predictions deterministic (Wang et al.) 2020 Li et al. 2021). However, our perspective on it differs we consider it from a causal standpoint. As a result, we arrive at the final loss function:
max CMI + ALELBO + >LENTs
(10)
where and y are seper-parameters that trade off the three loss functions. A graphical depiction of the proposed iT.CC-MSDA is shown in Figure
9

---

Published in Transactions on Machine Learning Research (04/2025)
6 Experiments
6.1 Experiments on Synthetic Data
138
10-
Mall
075-
5-
28 MD DMM
de
di
(a) Recovered ne
(b) Predicted y on the target segment
Figure 4: The Result on Synthetic Data. Due to the invariant conditional distribution p(y (d)),,,, even with the change of p(n,) as shown in Figure [(a] the learned p(y (n)) can generalize target segment in principle way
We conduct experiments on synthetic data to verify our theoretical results and the ability of the proposed method to adapt to a new domain. The synthetic data generative process is as follows: we divide the latent variables into 5 segments, which correspond to 5 domains. Each segment includes 1000 examples. Within each segment, we first sample the mean and the variance from uniform distributions [1,2] and [0.3, 1] for the latent exogenous variables n. and ng, respectively. Then for each segment, we generate Zes Zzd x and y according to the following structural causal model:
Ze ng, Ze := z + n., 2 MLP(Ze, 2.).
(11)
where following (Khemakhem et al 2020 we mix the latent ze and z, using a multi-layer perceptron to generate x. We use the first 4 segments as source domains, and the last segment as the target domain.
For the synthetic data, we used an encoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer, and a decoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer. We use a 3-layer fully connected network with 30 hidden nodes for the prior model. Since this an idcal environment to verify the proposed method, for paprr,,arametess, we set B heuristic constraints, and we set À = 1e- 2. Figure [(a) shows the true and recovered distributions of n. The proposed iLCC-LCS obtains the mean correlation coefficient (MCC) of 0.96 between the original n. and the recovered one. Due to the invariant conditional distribution p(y)ne), even with the change of p(n.) as shown in Figure 4(a) the learned p(y nn.) can generalize to target segment as depicted by the Figure 4(b)
and 7 = 0 to remove the
6.2 Experiments on Real Resampled PACS data
There exist datasets, such as PACS Li et al. 2017) and Office- Home (Venkateswara et al. 2017 commonly used for evaluating MSDA under previous paradigıns. These datasets exhibit very limited changes in label
ERM
MCDA
M35DA
LEC-MSDA
T-SVDeet
IWCDAN
LaCIM
IMSDA
iLCC-LCS (Ours)
Resampled PACS (Dec = 0.3)
Resampled PACS (Drc = 0.5)
Resampled PACS (Dxc = 0.7)
Figure 5: Classification results on resampled PACS data.
10

---

Published in Transactions on Machine Learning Research (04/2025)
distribution across domains. For instance, in the PACS dataset, the KL divergence of label distributions between any two domains is exceedingly small, approximately Dxc R 0.1. Consequently, these datasets are well-suited for evaluating models in the setting of conditional shift, where label distributions remain consistent across domains, as illustrated Figure I(b) In order to render the PACS dataset more suitable for assessing the adaptability of MSDA algorithms to more challenging and realistic scenarios characterized by significant label distribution shifts, we apply the filtering process. We resample the original PACS dataset, which contains domains, Photo, Artpainting, Cartoon, and Sketch, and shares the same seven categories. We filter the original PACS dataset by re-sampling it, i.e., randomly filter some samples in the original PACS to obtain pre-defined label distribution. As result, we can obtain three new datasets, resampled PACS (Dxc = 0.3), resampled PACS (Dxz 0.5), and resampled PACS (Dxz 0.7). Here Dxc 0.3(0.5, 0.7) denoles that KL. divergence of label distributions in any two different domains is approximately 0.3 (0.5, 0.7). Figure depicts label distributions in these resampled datasets.
0.0
Culta
100
Cardon S0ac3
Carboe
(b)
(c)
Figure 6: Label distributions of the resampled PACS datasets with DKC = 0.3 (a), 0.5 (b), 0.7 (c).
We compare the proposed method with state-of-the-art methods to verify its effectiveness. Particularly we compare the proposed methods with empirical risk minimization (ERM), MCDA (Saito et al. 2018).
M3DA (Peng al. 2019), LiC-MSDA (Wang et al.] 2020), T-SVDNet (Li et al. 2021), IRM (Arjovsky et al. 2019), IWCDAN (Tachet des Combes et al. 2020), LaCIM (Sun et al.) 2021]. b, iMSDA (Kong et al, 2022 SIG (Li et al. 2023), and MIEM Wen et al. 2024) In these methods, MCDA, M3DA, LIC-MSDA, T-SVDNet, and MIEM learn invariant representations for MSDA, while IRM, IWCDAN, SIG and LaCIM are tailored for label distribution shifts. All the methods above are averaged over 3 runs. All methods used the same network backbone, ResNet-18 pre-trained on ImageNet. Since it can be challenging to train VAE on high-resolution images, we use extracted features by ResNet-18 as our VAE input. We then use 2-layer fully connected networks as the VAE encoder and decoder, use 2-layer fully connected network for the prior model, and use 2-layer fully connected network to transfer n. Ze. For hyper-parameters,
set 8 4, y y= 0.1, x = le for the proposed method on all datasets. For the compared methods, we make every effort to tune their
to ensure a fair comparison.
The results by different methods on the resampled PACS are presented in Figure 5 Detailed results can be found in Tables 13 We can observe that as the increase of KT. divergence of label distribution, the perfor- mances of MCDA, M3DA, LtC-MSDA and T-SVDNet, which are based on learning invariant representations, gradually degenerate. In the case where the KL divergence is about 0.7, the performances of these meth are worse than traditional ERM. Compared with IRM, IWCDAN and LaCIM, specifically designed for label distribution shifts, the proposed iLCC--CS obtains the best performance, due to our theoretical guarantee for identifying the latent causal content variable, resulting in a principled way to guarantee adaptation.
6.3 Experiments on Real Terra Incognita data
We further evaluate the proposed iLCC-LCS on Terra Incognita dataset proposed in (Beery et al. 2018 used for evaluation for domain generalization. For Terra Incognita (Beory et al.] 2018). consists of 57, 868 images across 20 locations, each labeled with one of 15 classes (or marked as empty). Classes are either single species (e.g. "Coyote" or groups of species, egg. "Bird"). Figure shows the label distribution in
11

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Classification results on resampled PACS data (Dec = 0.3).
Methods
Accuracy
Art
+Cartoon
Photo
+Sketch
Average
ERM
82.3 +
81.3 +
94.9 + 0.2
76.2 = 0.7
83.6
MCDA Saito et
2018
76.6 + 0.6
85.1 + 0.3
96.6 + 0.1
70.1 + 1.3
82.1
M3SDA [eng eb al.
2019
79.6 t: 1.0
86.6 + 0.5
97.1 : 0.3
83.3 + 1.0
86.6
L+C-MSDA Wang
2020
82.7 + 1.3
84.9 = 1.4
96.9 = 0.2
75.3 = 3.1
84.9
T-SVDNet (Li al.
81.8 0.3
86.5 0.2
95.9 1 0.2
80.7 1 0.8
86.3
IRM Arjovsky al.
2019
79.6
77.0 1
L
71.7
IWCDAN (Tachet des Combes et
2020
84.0
78.1 = 0.7
96.0 =
75.5 - 1.9
83.4
LaCIM (Sun et al. 2021
63.1 1.5
72.6 1 1.0
82.7 + 1.3
71.5 1 0.9
72.5
iMSDA (Kng
2022
82.1 +: 1.2
86.1 0.3
96.9 1 0.2
80.6 1 1.2
86.4
STG [T.
2023
86.2 + 1.0
8633 0.8
97.3 + 0.2
73.3 + 0.8
85.7
MIEM Wen 2024
816 +
813
96.6
73.4
83.2
(LCC-LCS(Ours)
86.4 = 0.8
81.1
95.9
86.0 = 1.0
87.4
Table 2: Classification results on resampled PACS data (Dæg = 0.5).
Methods
Accuracy
Art
Cartoon
Photo
+Sketch
Average
ERM
85.41 0.6
76.4 + 0.5
94.4 1 0.4
85.0 0.6
85.3
MCDA Saito
2018
81.6 +
76.8
93.6 +
84.1 .6
84.0
Peng
81.2 + 1.2
77.5 = 1.3
94.5 =
84.3 0.5
84.4
LaC-MSDA Wang el al.
2020
85.2 + 1.5
75.2 + 2.6
94.9 + 0.6
85.1 + 2.7
85.1
T-SVDNet et ml. 2021
84.8 1 0.3
77.6 + 1.7
94.2 + 0.2
86.4 0.2
85.6
IRM (Arjovsky et al. 2019
81.5 + 0.3
71.1 = 1.3
94.2 = 0.1
78.7 = 0.7
0.7
81.4
IWCDAN (Tachet des Combes al.
2020
79.2 1.6
72.6 1 0.7
95.6 0.1
82.1 1.2
82.4
LaCIM (Sn al. 2021
67.4 1.6
1.6
66.6 = 0.6
81.0 1.2
82.3
74.3
iMSDA Kong
2022
82.4
76.4
94.3 0.2
84.1 0.6
84.3
STG [Li al.
79.3
95.2
80.1
85.1
MIEM Wen al.
2024
85.2 -
72.5 = 0.8
9 0.2
79.7 0.6
83.0
ILCCC- (.(CC))
89.0 + 0.7
77.6 + 0.5
95.0 + 0.3
87.4 + 1.6
87.3
Table 3: Classification results resampled PACS data (Drc 0.7).
Methods
Accuracy
Arr
-Cartoon
-Photo
-Skettch
Average
ERM
1
1 0.3
94.6 1
81.3 1
84.7
MCDA
Saito ct al.
2018
80.8 1 0.7
74.1 + 1.2
94.4 + 0.4
77.9 1 0.4
81.8
M3SDA Peng el al.
2019
82.7 + 1.3
76.2 + 1.0
94.5 + 0.7
80.8 + 1.2
83.6
LIC-MSDA Wang
2020
83.7 + 1.6
74.6 + 1.4
95.0 + 0.7
80.8 + 0.6
83.5
T-SVDNet [Li
2121
83.3 +
74.7 -
95.2 =
74.5 = 3.3
81.9
IRM Arjovsky al.
2019
+
69.4 +
80.3
IWCDAN (Tachel des Cumbes el
2020
76.3 0.8
73.9 + 1.6
93.1 + 0.5
77.6 + 3.8
80.2
LaCIM Sun et al.
2021
63.6 + 0.9
68.7 + 1.4
77.5 + 3.8
77.8 + 2.2
71.9
iMSDA Kong ol
2022
81.9 1 0.6
72.4 1.0
94.2 1 0.1
79.4 1 0.8
82.0
SIG [Li al. 2023
85.1 1 0.4
73.3 1 0.8
93.1 1
77.5 1 0.8
82.3
MIEM Wen
2024
85.8
94.2
CLCCLLC((Orr))
90.7 - 0.3
74.2 1
95.8 1 0.3
83.0 1 2.2
86.0
12

---

Published in Transactions on Machine Learning Research (04/2025)
raccoon
rabiit
coyote
Number of images
poy
cat
6000
emply
1000
squiree
dogg
300
bird
500
2000
skunk
redent
deer
badger
fox
Location
(a)
(b)
Figure 7: (a) Label distributions of the whole Terra Inoognita data. (b) Label distributions in the four domains of the Terra Incognita data, which are used in our experiments.
Table 4: Classification results TerraIncognice
Methods
Accuracy
-22
+L43
-L46
->77
Average
ERM
54.1 + 288
62.3
1 0.7
44.7
: 0.9
74.5 + 2.6
58.9
MCDA
Salto et al.
2018
54.9 + 4.1
61.2
1.2
42.7
+ 0.3
64.8
+ 8.1
55.9
M3SDA
(Pnng el al
2019
62.3 1.4
62.7
0.4
41.3
+ 0.3
57.4
+ 0.9
55.9
LAC-MSDA Wang et 8l 1020
51.9 5.7
516
1.3
45.7
1.0
09.1
0.3
55.3
T-SVDNet
58.2
61.9
0.3
45.6
68.2
IRM Arjovaky
57.5 1.7
60.72
424 0.6
74.1
1.6
58.7
IWCDAN (Tachel
Combes
2020
58.1 + 1.8
59.3
43.8+
58.9
3.8
55.0
LaCIM (Stn et al
2021
58.2 1. 3.3
59.8 1.6
46.3
70.8
1.0
58.8
iMSDA Kong et Bl
2022
59.3 + 1.3
62.9 0.8
47.1 + 1.0
70.4
1.0
59.9
SIG Li el al 2223
58.4 1..4:00 0.9
618 1 0.8
45.4 + 0.6
68.4
0.9
58.5
MIEM Wea et a.
2021
57.5 + 1.7
60.8 1 0.9
41.
0.8
67.9
1.1
57.6
SLCCCL.C5((oms))
64.3 3.4
63.1
1.6
44.7
0.4
80.8 +: 0.4
63.2
different locations. The label distribution long-tailed at each domain, and each domain has a different label distribution, hence it naturally has significant label distribution shifts, ideal for the challenging scenarios that LCS describes. We select four domains from the original data, L28, L43, L46, and L7, which share the same seven categories: bird, bobcat, empty, opossum, rabbit, raccoon, and skunk. Here 'L28' denotes that the image data is collected from location 28. Figure 7(b) depicts label distributions in the four domains above. Similar to implementation on PACS, for Terra Incognita data, we use the same network backbone, ResNet- 18, use 2-layer fully connected networks as the VAE encoder and decoder, use 2-layer fully connected network for the prior model, and use 2-layer fully connected network to transfer n. to Ze- For hyper-parameters, we set B = 0.1, X
4 for the proposed method on all datascts.
Table 4 depicts the results by different methods challenging Terra Incognit. The proposed iLCC-LCS achieves significant performance gain the challenging task ~L7. Among the methods considered, the proposed proposed ILCC-LCS and iMSDA stands out as the only two that outperforms ERM. This superiority is especially pronounced due to the substantial variation in label distribution across domains. In cases where the label distribution changes significantly, traditional approaches may lead to the development of minforrattve features, making them less effective for domain adaptation. contrast, the proposod method excels at capturing and adapting these label distribution changes, enabling accurate predictions even under such dynamic conditions.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Ablation study on resampled PACS data (Dez 0.7), and Terralncognita.
Methods
Accuracy oa resampled PACS data( - 0.7)
Accuracy on Terralnrouiia
+Art
-Cartoom
-Photo
-SSactch
Average
-L28
-+4i
LL.6
-L7
Avernge
LLCCLCS with 5-1
73.4 0.80
96.7
82.7 0.7
85.5
56.3 4.3
a 0.7
45.2
0.3
4 0.06
60.8
TCCELS with x=
0.
71.1 111111
96.5 1:
81.6 0.6
86.5
55.9 1:
50.4 0:66
45.0 1:000
79.6 0.05
60.0
LCC with =
81. 1.5
700 1 1.6
92.0 1 0.5
59.6 1 0.7
75.7
$48
58.9 1 1.8
46.8 1.4
73.1 1 0.6
56.4
LCCLes
90.7 = 0.3
7422077
95.3 z 0.3
83.0 = 22
86.0
643 2 3.4
63.1 a 1.8
44.7 0.
80.8 = 0.4
63.2
6.4 Ablation studies
Table displays the results of ablation studies conducted PACS (Dxc 0.3) and TerraIncognita, respectively. Notably, we observe significant improvement in performance (approximately 10% and 5%, respectively) for the proposed method on both datasets when employing entropy regularization (as per Eq. g This finding aligns with previous research (Wang et al. 2020 Li et al. 2021), which has also underscored the importance of entropy regularization (Eq. (9. From the viewpoint the proposed causal model, entropy regularization essentially encourages causal influence between y and ney as elaborated upon in Section 5 Furthermore, the hyperparameter 3 plays a pivotal role in enhancing performance by enforcing independence among the latent variables n4.
We further consider the sensitive analysis for 8, À and 7, which are shown in Figure 8 on Terra data. There results reveal that the model's accuracy is most sensitive to changes in 7. As increased from to 0.10, accuracy improved, peaking at 63.2%, before decrcasing at higher values. Similarly, 3 showed a slight improvement up to 4, with the highest accuracy at 63.2%, and a small decline beyond that. For X, accuracy increased with X, but exhibited more volatility, with the best performance at À
The sensitivity of the parameter y our model aligus with common trends observed in domain adaptation methods Wang al. 2020); Li et al. (2021) Kong et al. 2022). As "y controls the conditional entropy in the target domain, it plays crucial role in making label predictions more deterministic. Our results suggest that, similar to many existing methods in domain adaptation, small increases in initially improve the model's accuracy, as better aligns the distribution between source and target domains. However, beyond an optimal value, further increases in y introduce overfitting or distortions in the representation, leading decrcase in performance. This highlights the importance of carcfully tuning y for achieving the best performance, as it a sensitive parameter influencing both the accuracy and the generalizability of the model across domains.
Accurccy
Ascuracy
2.0
3.5
(a.)
(b)
(c)
Figure 8: Scnsitive analysis for B, À and y
6.5 t-SNE visualization
We visualize features of each class obtained by the proposed method via I-SNE to show how the distributions of learned features change across domains. Figure g (a)-(d) depicts the detailed distributions of the learned features by the proposed method different domains of TerraIncognita data. Differing from the methods that learn invariant representations across domains, the distributions of the learned features by the proposed method change across domains.
14

---

Published in Transactions on Machine Learning Research (04/2025)
10-
10-
06-
0.
0.4
0.0
00-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.
0.8
(a)
0.8
0.8
0.
0.2
02
0.0
0.4
0.6
02
(c)
(d)
Figure The t-SNE visualizations learned features n. of different domains on the +L7 task Ter- raIncognita. (a)-(d) The learned features n. in domain L28, L43, L46, L7. Here 'L28' denotes that the image data collected from location 28. We observe that the distribution of learned feature n. by the proposed method changes across domains.
15

---

Published in Transactions on Machine Learning Research (04/2025)
7 Further Discussion
7.1 The Proposed Causal Graph
The versatility our proposed causal graph extends beyond being narrowly designed for domain adaptation. Instead, we hope that it potential to effectively tackle a wide range of tasks across diverse problem domains. In scenarios such as causal or anti-causal tasks, where a more specific causal graph structure is deemed essential, our proposed causal graph might stand out may serve not only as solution for domain adaptation but also flexible and adaptable framework. It source of inspiration for various tasks. This adaptability is rooted in the intentional modeling of latent causal relationships within unstructured observational data. To illustrate, consider its application in segmentation tasks, where interpreting graph nodes as distinct regions. Here, the absence of direct connections between nodes may be reasonable; instead, connections should be contemplated within high-level latent variables. This attribute enhances the versatility and potency of the proposed causal graph, making it adaptable the nuanced requirements of diverse tasks and problem domains.
7.2 Limitations
One of the foundational assumptions mandates significant. alterations in the latent noise variables, driving observable variations in the data. However, the pragmatic fcasibility of moeting this assumption in real- world applications adds a layer of intricacy. The endeavor to induce substantial changes in latent noise variables encounters challenges rooted in the inherent complexities of data-generating processes. Specifi- cally, in practical applications, the availability of extensive pool of training data across diverse domains may be limited. This limitation introduces a potential vulnerability for the proposed method. Addressing such scenarios involves effectively leveraging the independence among n. For instance, the imposition of independence can be achieved through various regularization techniques, as enforcing a hyperparameter to enhance the independence among n; in Eq. 7
7.3 Model Selection
Model selection in domain adaptation is challenging due to the complexities of transferring knowledge across domains with different distributions. Unlike traditional supervised learning, where training and test data share the same distribution, domain adaptation requires models to generalize from a source domain to a target domain, often with limited labeled data. Selecting the right model and tuning hyperparameters for domain alignment and task performance is difficult, especially with large domain shifts or noisy target data. Additionally, the choice of adaptation techniques (e.g., feature aligument, adversarial training) depends on factors like domain discrepancy, requiring empirical testing. The lack of universal framework further complicates model selection, as it is highly problem-specific and influenced by domain similarity and data quality.
In domain adaptation, due to the access to input data in the target domain, the community often resorts to model selection based on the target domain's characteristics, assuming that since input data is available, human labeling can be used to generate labeled samples for model selection. However, first, manually labeling data the larget domain can be prohibitively expensive, time-consuming, and labor-intensive, especially for complex or large-scale datascts. Sccond, relying on target domain labeling for model selection undermines one of the key objectives of domain adaptation: minimizing the need for labeled data in the target domain. Additionally, even when labels are available, ensuring that the model trained these labeled samples generalizes well to unseen data is challenging.
In the future, providing unified framework for evaluation in domain adaptation remains an interesting and valuable goal. Such a framework could standardize the assessment domain adaptation methods, offering clear metrics for comparing different approaches across various domains and tasks. This would help address challenges like domain discrepancy, generalization, and model robustness, allowing researchers to better understand the strengths and weaknesses of different techniques.
16

---

Published in Transactions on Machine Learning Research (04/2025)
8 Conclusion
The key for domain adaptation is to understand how the joint distribution of features and label changes across domains. Previous works usually assume covariate shift or conditional shift to interpret the change of the joint distribution, which may be restricted in some real applications that label distribution shifts. Hence, this work considers a new and milder assumption, latent covariate shift. Specifically, we propose a latent causal model to precisely formulate the generative process of input features and labels. We show that the latent content variable in the proposed latent. causal model can be identified up to scaling. This inspires new method to learn the invariant label distribution conditional the latent causal variable, resulting in a principled way to guarantee adaptation to target domains. Experiments demonstrate the theoretical results and the efficacy of the proposed method, compared with stale-of-the-arl methods across various data sets.
References
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Chan-Chisttpphe Gagnon-Audel, Yoshua Bengio, Ioan- nis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for oustof-distribution generalization. Advances in Neural Information Processing Systems, 34:3438 3450, 2021.
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D Amour, Arthur Gretton, Sanmi Koyejo, Matt J Kusner, Stephen R Pfohl, Olawale Salaudoon, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In International Conference on Artificial Intelligence and Statistics, pages 9637-9661. PMLR, 2023.
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottle- nock. arxi preprint 161im:612. 22.00410 10, 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint Xix: 028899 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning, pages 81-88, 2007.
Victor Bouvier, Philippe Very, Céline Hudelot, and Clément Chastagnol. Hidden covariate shift: A minimal assumption for domain adaptation. arXiv preprint wrXiv:1907.12299 2019.
Yichao Cai, Yuhang Liu, Zhen Zhang, and Javen Qinfeng Shi. Clap: Isolating content from style through contrastive learning with augmented prompts. In European Conference on Computer Vision, pages 130- 147. Springer, 2024.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A cansal frame- work for distribution generalization. IEFE Transactions on Pullern Analysis and Machine Intelligence, 2021.
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia Vogt. Identifiability results for multimodal contrastive learning. ICLR, 2023.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dodain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
Muhammad Ghifary, David Balduzii, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions on pattern analysis and machine intelligence, 39(7):1414-1438 2016.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. Tn International conference. on machine learning, pages 2839 2848. PMLR, 2016.
I. Higgins, Loic Matthey, A. Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. bela-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlincar causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopl, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 559-868. PMLR, 2019.
Tyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear iea: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pages 2649 2658. PMLR, 2018.
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes arXiv preprint arXiv:1312.6114 2013.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michíhiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in- the-wild distribution shifts. In International Conference on Machine Leurning, pages 5637 5664. PMLR, 2021.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang, Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pages 11455-114 11472. PMLR, 2022.
Meelis Kull and Peter Flach. Patterns of dataset shift. In First International Workshop on Learning over Multiple Conterts (LMCE) at ECML-PKDD, 2014.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gen- eralization. Tn Proceedings of the IEEE international conference on compuler vision, pages 5542-5550, 2017.
Ruihuang Li, Xu Jia, Jianzhong Hc, Shuaijun Chen, and Qinghua Hu. T-svdnct: Exploring high-order pro- tolypical correlations for multi-source domain adaptation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 9991-10000, 2021.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via condi- tional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. Subspace identification for multi-source domain adaptation. Advances Neural Information Processing Systems, 36:34504-34518, 2023.
Zachary Lipton, Yu- Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122 3130. PMT.R 2018.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tic- Yan Liu. Learning causal semantic representation for out-of-distributiin prediction. Advances in Neural Information Processing Systems, 34, 2021.
Yuhang Liu, Wenyong Dong, and Mengehu Zhou. Frame-based variational bayesian learning for independent dependent source separation. IEEE Transactions Neural Networks and Learning Systems, 29(10): 4983-4998 2018.
Yuhang Liu, Wenyong Dong, Lei Zhang, Dong Gong, and Qinfeng Shi. Variational bayesian dropout with a hierarchical prior. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recog- nition, pages 7124 7133, 2019.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight-variant latent causal models. arXiv preprint orXiv:2208. 14153, 2022.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable latent neural causal models. arXiv preprint orXiv:2403. 15711, 2024a.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable latent polynomial causal models through the lens of change. In The Twelfth International Conference on Learning Representations, 2024b.
Yuhang Liu, Zhen Zhang, Dong Gong, Biwei Huang, Mingming Gong, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Revealing multimodal contrastive representation learning through latent partial causal models. XrXiv preprint arxi:2240..0 06229, 2024c.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant, causal repre- sentation lcarning for out-of-distribution generalization. In International Conference on Learning Repre- sentations, 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In Inter- national Conference on Machine Learning, pages 7313-7322 PMLR, 2021.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Boosting domain adaptation by discovering latent domains. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3771-3780, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVP international conference on compuler vision, pages 1406-1411 1415, 2019.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7854):947--112, 2016.
Kuniaki Saito, Kohci Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pages 3723-377 2018.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Fleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv:1206.64 2012.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-fow networks (gin). arXie preprint arXiw:2001.04: 04872, 2020.
Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang. Data-driven approach multiple-souree domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3487-3496. PMLR, 2019.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adap- tation with invariant representation learning: What transformations to learn? Advances in Newral Infor- mation Processing Systems, 34, 2021.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural informalion processing systems, 20, 2007.
Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tie- Yan Liu. Recovering latent causal factor for generalization to distributional shifts. Advances in Newral Information Processing Systems, 34, 2021.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33:19276 19289, 2020.
Victor Veitch, Alexander D' D'Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. Advances in Neural Information Processing Systems, 34:16196- 16208, 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018 5027-00777 2017.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervieed learning with data augmentations provably isolates content from style. NewrIPS, 34:16451 16467, 2021.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang, Learning to combine: Knowledge aggregation for multi-source domain adaptation. In European Conference on Computer Vision, pages 727-744. Springer, 2020.
Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribation generalization with causal invariant transformations. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 375 385, 2022a.
Zengmao Wang, Chaoyang Zhou, Bo Du, and Fengxiang He. Self-pacers supervision for multi-source domain International Joint Conference on Artificial Intelligence,
adaptation. In Proceedings of the IJCAI-22, 2022b.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pages 631-639. PMLR, 2014.
Lisheng Wen, Sentao Chen, Mengying Xie, Cheng Liu, and Lin Zheng. Training multi-source domain adaptation network by mutual information estimation and minimization. Neural Networks, 171:353-38 361 2024.
Luyu Yang, Yogesh Balaji, Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In European Conference on Computer Vision, pages 608-624. Springer, 2020.
Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint ar 1an::22552599 2012.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference Machine Learning, pages 819-827, PMLR, 2013.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: causal view. In T Twit AAAI conference on artificial intelligence, 2015.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Zhen Zhang, Ignavier Ng, Dong Gong, Yuhang Liu, Ehsan Abbasnejad, Mingming Gong, Kun Zhang, and Javen Qinfeng Shi. Truncated matrix power iteration for differentiable dag learning. In Advonces in Neural Information Processing Systems, 2022.
Zhen Zhang, Ignavier Ng, Dong Gong, Yuhang Liu, Mingming Gong, Biwei Huang, Kun Zhang, Anton den Hengel, and Javen Qinfeng Shi. Analytic dag constraints for differentiable dag learning. arXiv preprinl ar Xiv:25003 19218, 2025.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in neural informalion processing systems, 31, 2018.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representa- tions for domain adaptation. In International Conference on Machine Learning, pages 7523-7532. PMLR, 2019.
Sicheng Zhao, Bo Li, Penglei Xu, Xiangyu Yue, Guiguang Ding, and Kurt Keutzer. Madan: multi-source adversarial domain aggregation network for domain adaptation. International Journal of Computer Vision, (8)(33233 2424, 2021.
21

---

Published in Transactions on Machine Learning Research (04/2025)
.1 The Proof of Proposition 4.1
To establish non-identtfiabiliy,, it sulfices to demonstrate the existence of an alternative solution that differs from the ground truth but can produce the same observed data. Let's consider the latent causal generative model defined in Eqs. | to as our ground truth, represented in Figure 2(a) where there exists a causal relationship from Ze to Zg- Now, we can always construct new latent variables z' as follows: z = ge (e(n.) and x ns: depicted in Figure 2(b] Importantly, there no causal influence from z to zz in this construction, which different from the ground truth in Figure 2(a) It becomes evident that this new set of latent variables z' can generate the same as obtained by z through composition function f fo where f((2) = I/': 8s2 8x2 (g.1 + 2[)]- This scenario leads to non-identifiable result, as the same observed data can be produced by different latent variable configurations, thus confirming non-identifiabiitt.
.2 The Proof of Proposition 4.2
For convenience, we first introduce the following lemma.
Lemma Denote the mapping from n to z by g. given the assumption (ti) in proposition 4.2 that the mapping from a to x is invertible, we have that the mapping from n to x, e.g., fo g, is invertible.
Proof can be easily shown by the following: since the mapping from n to z is defined as 2 where both ge and 8.2 are assumed to be invertible, we can obtain the inverse function of the mapping g from n to z as follows: n. = =g==g-lo (zc). ng = 8.2 (z.) g.i (ze), which clearly shows that the the mapping is invertible. Then by the assumption (ii) in proposition 4.2 that f is invertible, we have that the composition of f and g is invertible, Le., fo tog is invertible.
The proof of proposition 4.2 is done in three steps. Step I is to show that the latent noise variables n be identified up to linear transformation, n Ań + c, where ñ denotes the recovered latent noise variable obtained by matching the true marginal data distribution. Step II shows that the linear transformation can be reduced to permutation transformation, i.e., n d-separation criterion in the graph structure in Figure ( i.e., y is dependent on n. but independent of nf, given u.
(()) c. Step III shows that z, = h(2.), by using
Step I: Suppose we have two sets of parameters 0 (f. T) and 0 (f.g. T, ?) corresponding to the same conditional probabilities, ie., P(f.g T, (x|u) = P(...g.f.m)) (xlu) for all pairs (x, u). Since the mapping from to is invertible, as Lemma [ with the assumption (i), by expanding the conditional probabilities (see Step I for proof of Theorem in Khemakhem et al. (200 for more details), we have:
log det J (fog)) (x)) log P(T,T) (n|u) = log det J (fog) (x) log P:T. (f)
(12)
Using the exponential family Eq. to replace P(T...)) (n|u), we have:
log det (fog)) (x) T (f gx) (x)) - log II Zu( =
log det J (x) -. ((f g) (x)) - log II 2u(,,
(13)
Then by expanding the above at points u, and un mentioned in assumption (iii), and using Eq. 13 at point subtract Eq. 13 at point up, we find:
(T(m), (u))) + Σ log Z,(w)) Z,(u:)
(T()), log Zi(wo) 2,(u)
(14)
Here 77(u) m7()) - (un)) Then by combining the 28 expressions (from assumption (iii) we have 20 such expressions) into a single matrix equation, can write this in terms of L from assumption (iii).
LTT(n)= (T + b.
(15)
22

---

Published in Transactions on Machine Learning Research (04/2025)
Since L is invertible, can multiply this expression by its inverse from the left to get:
T(n) (f(
(16)
where A = (L" According to a lemma 3 in Khemakhem et al. (2020) that there exist & distinct values n, nf such that the Derivative ('.))..... (n.) are linearly independent, and the fact that each component of Ti is univariate. We can show that A is invertible.
Step II Since assume the noise to be two-parameter exponential family members, Eq. expressed as:
16 can be re-
Ti(n) T2(n) ) = A ( Tn T2(n) ) + c.
(17)
Then, we re-express T2 in term of 11 e.g., T2(m))= = (TT(()))))) where is nonlinear mapping, As result, we have from Eq. 17 that: (a) Ti (T:)) can be linear combination of (f) and T2(û), and (b) ((T) (n:) can also be linear combination of T, (n) and f2((.. This implies the contradiction that both Ti (14))) and its nonlinear transformation t(T1 (n:)) can be expressed by linear combination of Í, (f) and T2) This contradiction leads that A can be reduced to permutation matrix P (See APPENDIX C Sorrenson et al. (020)) for more details):
(18)
where P denote the permulation matrix with scaling, e denote a constant vector. Note that this result holds for not only Gaussian, but also inverse Gaussian, Beta, Gamma, and Inverse Gamma (See Table in Sorrenson et al. (2020)).
Step III: The above result shows that can obtain the recovered latent noise variables ñ up permutation and scaling transformation of the true which are learned by matching the true marginal data distribution p(x|u). However, it still not clear which part of á corresponds to ney c.g,, permutation indeterminacy. Fortunately, the permutation can be removed by d-separation criteria in the graph structure in Figure ((a) which implies that n, is dependent on label y, while n, independent with y, given u. Thus, together with the d-separation criteria, Eq. 18 implies that:
n Peñe ce.
(19)
Then by model assumption as defined Eq. using 2, to replace n. obtains:
ge (zc) P P.g. (2c) + Ce.
(20)
which can re-expressed as:
2. ga (P. (2.) - c.)) h(((2));
(21)
where h is clearly invertible, which implies that z, can be recovered up to block identifiability.
23


Reviews:
[{"review_id": "TKjjscTz2m", "reviewer": "Reviewer_opGo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this work is that it is clearly motivated, theoretically justified, and provides experimental results on two difficult datasets from the DomainBed suite. It is very challenging to perform better than ERM on these datasets.\n\nThe critical weakness of this work is that it involves tuning three hyperparameters $\\beta$, $\\lambda$, and $\\gamma$, and makes no mention of model selection. As argued in https://arxiv.org/abs/2007.01434, domain generalization (also applies to domain adaptation) algorithms should be rendered incomplete unless they offer a sensible way to tune the hyperparameters without accessing the test set labels. I suspect the authors' approach is particularly sensitive to the values of $\\lambda$ and $\\gamma$. The important topic of model selection is swept under the rug with the following statement: \"for hyper-parameters, we set $\\beta = 4$, $\\gamma = 0.1$, $\\lambda = 1e − 4$ for the proposed method on all datasets.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a domain adaptation algorithm that assumes p(y|zc) is invariant, and that p(zc) can shift. This weakens the  assumption of invariance of p(y) across domains that is often assumed by existing approaches. They provide block-identification results for zc, and demonstrate the efficacy of their algorithm on a synthetic data, as well as two DomainBed datasets that have been resampled to induce a shift in p(y) between domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It may not be possible to formulate a sensible way to perform model selection with no prior assumptions on the nature of distribution shift between the training and test domains. I suggest that you discuss the difficulty of model selection, and include an ablation study that shows that your algorithm is fairly robust (or not) to the choice of these hyperparameters. Leaving this issue undiscussed gives the false impression that this is not an important issue in this line of work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iqCVc3i336", "reviewer": "Reviewer_8eWb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths of the Paper**  \n- The causal model for latent covariate shift is novel and intriguing. The introduction of the latent variable model provides a more flexible framework for capturing the variability component of the distribution.  \n- The theoretical analysis demonstrates the identifiability of the latent variable $\\mathbf{z}_c$.  \n- Experimental results show that the proposed method outperforms the baselines on both synthetic and real-world datasets.  \n\n**Weaknesses of the Paper**  \nThe main concerns about the paper are the lack of clarity regarding the motivation behind the latent model, its theoretical properties, and the implementation details of the proposed method.  \n\n- **Clarity on Relationship between Figures 1(c) and 2(a):** The relationship between the causal model in Figure 1(c) and Figure 2(a) is unclear. While Figure 2(a) seems to be a more detailed realization of Figure 1(c), the motivation and benefits of introducing the variables $\\mathbf{n}_c$ and $\\mathbf{n}_s$ are not well explained. This section could be improved with a more thorough discussion of the advantages and necessity of adopting a more complex causal model.  \n\n- **Theoretical Properties of the Proposed Method:** Although Proposition 4.2 establishes that the latent variable $\\mathbf{z}_c$ is identifiable by matching the true marginal data distribution, it is unclear how the proposed loss function in Eq. (10) facilitates this. In Eq. (10), several approximations and heuristic regularizers are introduced, but their role in recovering $\\mathbf{z}_s$ remains unexplained. Providing a detailed discussion on how the proposed method aligns with the theoretical guarantees would strengthen the paper.  \n\n- **Implementation Details:** The superior performance of the proposed method appears to depend heavily on the entropy term, yet the process for generating the estimated label $\\hat{y}^{\\mathcal{T}}$ is not well-introduced. Additionally, it is unclear how the quality of these estimated labels affects the performance of the method. More comprehensive details on this aspect would enhance the clarity of the implementation.  \n\n- **Parameter Selection:** Different parameter setups for $\\lambda$, $\\beta$, and $\\gamma$ are used across datasets, but the method for determining these parameters is not discussed. Including a more detailed explanation of the parameter selection process or conducting sensitivity analyses, would improve the robustness and reproducibility of the experimental results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates multi-source domain adaptation, where the learner observes data from multiple source domains and aims to train a classifier that performs well on an unlabeled target domain. The main contribution is the introduction of a new causal model that extends the covariate shift framework by incorporating a latent content variable $\\mathbf{z}_c$, which serves as a common cause of both the feature $\\mathbf{x}$ and the label $y$. From a theoretical standpoint, the paper analyzes the identifiability of $\\mathbf{z}_c$ under specific assumptions. Building on these insights, the authors propose a variational autoencoder–based method. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed approach.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Another important distribution problem is (generalized) target shift [1,2]. It would be valuable for the authors to discuss the relationship between the proposed model and the label shift problem.  \n- Could you provide a more detailed discussion of the relationship between the causal model in Figure 1(c) and Figure 2(a)?  \n- Could you include an analysis of how the proposed method recovers the latent variable $\\mathbf{z}_c$?  \n- Could you add more details about the process of producing the estimated label $\\hat{y}^\\mathcal{T}$, as well as how the parameters $\\lambda$, $\\beta$, and $\\gamma$ are selected? Additionally, please provide a discussion on how these parameters affect the performance of the method.  \n\n[1] Zhang et al. Domain adaptation under target and conditional shift. In ICML 2013.\n\n[2] Tachet des Combes et al. Domain adaptation with conditional distribution matching and generalized label shift. In NeurIPS 2020.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HBrCc1iqoo", "reviewer": "Reviewer_FUfs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ Overall, the theoretical insights have been well clarified. For instance, this paper provides a theoretical assurance for recovering the latent cause of the label variable.\n+ The experiments can verify the effectiveness of some parts of the proposed method, with visualization of feature distribution provided for a better understanding of the proposed method.\n+ The overall performance is better than the methods before 2021 on various datasets with label distribution shifts.\n\nCons:\n\n1. The key idea of this paper is very similar to iMSDA [A], both assuming that $p(y|z_c)$ is invariant. To better clarify the contribution of this paper, please compare it with [A] comprehensively.\n[A] Partial Identifiability for Domain Adaptation. ICML 2022.\n\n2. The latest methods from 2022-2024 should be compared methods, particularly iMSDA [A]. Please also feel free to choose some methods from 2023 and 2024 to compare with.\n\n3. The ablation study should include at least one more experiment with $\\lambda$=0 so that the effectiveness of $L_{ELBO}$ can be verified.\n\n\n4. Please evaluate the sensitivity of $\\beta$ in Eq. (7) and $\\lambda, \\gamma$ in Eq. (10).\n\n5. Are the experimental settings, e.g., hyper-parameters and implementation of encoder/decoder, on PACS the same as on TerraIncognita? If not, please provide more details on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a latent covariate shift (LCS) paradigm to deal with covariate shift, conditional shift, and label distribution shift in Multi-Source Domain Adaptation (MSDA). Within this paradigm, this paper presents an intricate causal generative model by introducing latent noises across domains, along with a latent content variable and a latent style variable to achieve a more nuanced rendering of observed data. The proposed method learns the label distribution conditioned on the identifiable latent content variable, thereby accommodating more substantial distribution shifts. Experimental results show the effectiveness of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the questions in the Cons and revise the manuscript accordingly.", "broader_impact_concerns": "There is no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new paradigm called latent covariate shift (LCS), which breaks the assumption of consistent label distribution across domains. The authors then proposed a new algorithm accordingly. Reviewers recognized the contributions of this work but also raised some concerns, such as implementation details, hyperparameter setting, connections to existing work, etc. The authors have provided very detailed responses as well as additional experimental results, which have been incorporated into the revised paper.
Decision: Accept (poster)