Identifying Latent Causal Content for Multi-Source Domain Adaptation
Authors: Yuhang Liu
Zhen Zhang
Dong Gong
Mingming Gong
Biwei Huang
Kun Zhang
Javen Qinfeng Shi
Venue: Submitted to ICLR 2023 (2023)

Abstract: Multi-source domain adaptation (MSDA) learns to predict the labels in target domain data, under the setting that data from multiple source domains are labelled and data from the target domain are unlabelled. Most methods for this task focus on learning invariant representations across domains. However, their success relies heavily on the assumption that the label distribution remains consistent across domains, which may not hold in general real-world problems. In this paper, we propose a new and more flexible assumption, termed \textit{latent covariate shift}, where a latent content variable $\mathbf{z}_c$ and a latent style variable $\mathbf{z}_s$ are introduced in the generative process, with the marginal distribution of $\mathbf{z}_c$ changing across domains and the conditional distribution of the label given $\mathbf{z}_c$ remaining invariant across domains. We show that although (completely) identifying the proposed latent causal model is challenging, the latent content variable can be identified up to scaling by using its dependence with labels from source domains, together with the identifiability conditions of nonlinear ICA. This motivates us to propose a novel method for MSDA, which learns the invariant label distribution conditional on the latent content variable, instead of learning invariant representations. Empirical evaluation on simulation and real data demonstrates the effectiveness of the proposed method.

Under review as a conference paper at ICLR 2023
IDENTIFYING LATENT CAUSAL CONTENT FOR MULTI- SOURCE DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-source domain adaptation (MSDA) learns to predict the labels in target do- main data, under the setting that data from multiple source domains are labelled and data from the target domain are unlabelled. Most methods for this task focus on learning invariant representations across domains. However, their success re- lies heavily on the assumption that the label distribution remains consistent across domains, which may not hold in general real-world problems. In this paper, we propose new and more flexible assumption, termed latent covariate shift, where a latent content variable 2, and a latent style variable 2g are introduced in the gen- erative process, with the marginal distribution of z, changing across domains and the conditional distribution of the label given 2, remaining invariant across do- mains. We show that although (completely) identifying the proposed latent causal model is challenging, the latent content variable can be identified up to scaling by using its dependence with labels from source domains, together with the identifi- ability conditions of nonlinear ICA. This motivates us to propose a novel method for MSDA, which learns the invariant label distribution conditional on the latent content variable, instead of learning invariant representations. Empirical evalua- tion on simulation and real data demonstrates the effectiveness of the proposed method.
INTRODUCTION
Traditional machine learning requires the training and testing data to be independent and identically distributed [Vapnik T999). This strict assumption may not be fulfilled in various potential real-world applications. For example, in medical applications, it is common to seek to train model on patients from a few hospitals and generalize it to a new hospital (Zech et al. 2018). In this case, it is often reasonable to consider that the distributions of data from training hospitals are different from the new hospital (Koh et al.) 2021). Domain adaptation is a promising research area to handle such problems. In this work, we focus on multi DA (MSDA) settings where source domain data are collected from multiple domains. Formally, let x denote the input, e.g. image, y denote the labels in source and target domains, and D denote the domain index. We observe labeled data pairs (x yS) from the multiple joint distributions p(x, y D = 1). ....,( y D = m...... ..., p(x, y D = M) in source domains, and unlabeled input data samples x' from the joint distribution p(x, y D in the target domain. The training phase of MSDA is to use the sets of (xS,y') and x" to train a predictor so that it can provide a satisfactory estimation for y in the target domain. The key for MSDA is to understand how the joint distribution Pa [x, y) change across all different source and target domains.
=m))
Most early methods assume that the change of the joint distribution results from Covariate Shift (Huang et al. 2006 Bickel et al. 2007 Sugiyama et al. 2007 Wen et al. 2014), e.g., Pr (x.(x,)))
Po (y(())pp (). as depicted by Figure ((a) This setting assumes that Po changes across domains, while the conditional distribution pa (y]x) is invariant across domains. Such assumption may not always hold for some real applications, e.g., image classification. For example, the assumption of invariant ()(()) implies that (y) should change as Po (x) changes. However, we can easily change style information (e.g., hue, view) in the images to change Po (x()) and keep (y( unchanged, which is common in classification but violates the assumption.
In contrast to covariate shift, most current works consider Conditional Shift as depicted by Figure i(b) It assumes that the conditional pr (xly) changes while Po (y) is invariant across domains

---

Under review as a conference paper at ICLR 2023
Zhang et al. 2013
2015
Schölkopf al. 2012 Stojanov et al.
2021 Peng al. 2019). This
situation motivates a popular class of methods focusing on learning invariat representations across domains to approach the latent content variable z. in Figure [(b) Ganin et al. 2016 Zhao al. 2018 Saito et al. 2018 Mancini et al., 2018 Yang et al. 2020 Wang et al. 2020: Li et al., 2021
Stojanov et al. 2021). However, the label distribution Pa y)) may change across domains in many
real application scenarios (Tachet des Combes et al.) 2020; Lipton et al. 2018 Zhang et al., 2013). for which learning invariant representations may be resulting degenerating performance. In the- ory, there exists an upper bound on the performance of learning invariant representations when label distribution changes across domains (Zhao al. 2019).
D
Z
Ze
(a) Covariate Shift
(b) Conditional Shift
(c) Latent Covariate Shift
Figure I:
The illustration of three different assumptions for MSDA. (a) Covariate Shift: po(x) changes
across domains, while Po x) invariant across domains. (b) Conditional Shift: Po(y) invariant, while Po (xly) changes across domains. (c) Latent Covariate Shift: Po(z.) changes across domains while pa (y (a) is invariant across domains.
In some real-world scenarios, the label distribution po (y) and the corresponding pa. (Ze) may vary across domains. For example, image classification, 2, represents the essential visual semantic features for discriminating classes (e.g., fur texture color for animal classification) may vary across domains (e.g., geographical location) but are also stably associated to specific domains. In such scenarios, if all domain variant features deciding y. On the other hand, since such varying features are associated with and covariant with the domain, they can be usable and useful in MSDA. To handle this scenario, assumption, Latent Covariate Shift (LCS), as depicted by Figure [(c) Unlike the conditional shift, LCS assumes that there is latent content variable ze, whose distribution po (Ze) changes across domains. Meanwhile the label conditional distribution Po ze) is invariant. By combing these assumptions, LCS enables the label distribution to change across domains.
excluded from Ze, Ze will be non-informative for
propose new
To understand more deeply and handle LCS, we propose latent causal model to formulate the data and label generating process, by introducing the latent style variable z, to complement z, as depicted in Figure 2 To analyse the identifiability of the proposed causal model, we also introduce latent noise variables n, and n2, which represent some unmeasured factors influencing z, and z,, respectively. As result, we can leverage recent progress in the identifiability result of nonlinear ICA (Hyvarinen et al.) 2019, Khemakhem et al. 2020) to analyse the identifiability of the proposed latent causal model. We show that although completely identifying the proposed latent causal model is often not possible without further assumptions due to transitivity in latent space, partially identi- fying the latent content variables 2, up scaling is tractable, by integrating the identifiability result of nonlinear ICA with the dependence between n, and y. This motivates us to propose novel method learn the invariant conditional distribution Pa (ylze) for LCS, instead of learning invari- ant representations. Relying on the guaranteed identifiability on Ze, the proposed method provides principled way to ensure that the covariant Z can be identified on the target domain data, and the leamned predictor Pa (y zz)) can generalize to the target domain. Empirical evaluation on synthetic and real data demonstrates the effectiveness of the proposed method, compared with sfte-of-the-ar methods.
Overall, our main contributions can be summarized as follows: (i) Differ from the commonly-used Conditional Shift as shown in Figure (b), which assume label distribution to be the same across domains, we propose a new problem setting, latent covariate shift, as shown in Figure (c). (ii) We propose latent causal model for latent covariate shift. Leveraging the existing identifiability results of nonlinear ICA, we provide an analysis about the identifiability of the proposed latent causal graph, which provides guarantee for identifying the latent causal content variable Ze. (iii) Under the identifiability result, we design a new method for domain adaptation, and empirically evaluate the proposed method on simulation and real data.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Learning invariant representations. Due to the limitation of covariate shift in image data, most current works for domain adaptation consider the conditional shift, which learns invariant represen-
tations across domains (Ganin et al. 2016, Zhao et al. 2018: Saito et al., 2018 Mancini et al. 2018
Yang et al., 2020 Wang et al. [020 Li et al. |021 Wang et al. 2022b] Zhao et al., 2021). Such invariant representations can be obtained by applying suitable linear nonlinear transformations on the input data. The key of these methods is how to enforce the invariance of the learned representa- tions. For example, the invariance 2018), or by a domain discriminator for adversarial training (Ganin et al. [016)] Zhao et al., [0188 2021 or by moment matching (Peng al. 2019), or by relation alignment loss (Wang al.. 2020)
be enforced by maximum classifier discrepancy (Saito al.
or by pseudo labeling (Wang et al.] 2022b). However, all these methods require label distribution to be invariant across domains. As result, when label distribution is varying across domains, they may perform well only in the overlapping areas among label distributions in different domains, while facing with challenges in the non-overlapping areas. To overcome this, recent progress focuses on learning invariant representations conditional on the label across domains (Gong et al.. 2016 Ghi- fary al. 2016; Tachet des Combes et al., (020). One of the challenges in these methods is that the labels in the target domain is unavailable. More importantly, these methods do not guarantee that the learnt representations to be consistent with the true relevant information for predicting in the target domain, thus there is no principled way to guarantee that the learned predictor can generalize to the target domain.
Learning invariant conditional distribution Po (y(y)z)). There exist few of works exploring the invariant conditional distribution Po (y(ze) for domain adaptation (uull & Flach 2014 Bouvier ett al., 2019). Differ from these two works, the proposed method provides the identifiability of Ze, so that the learned p., (y(z.) in this work can generalize to the target domain in a principled way.
Besides, in the context of out-of-distribution generalization, some recent works explore learning invariant conditional distribution Pa (y 22) (Arjovsky et al. 2019 Sun et al. 2021 Liu et al. 2021
Lu et al. 202I). For example, Arjovsky et al. (2019) imposes learn the optimal invariant predictor across domains from the viewpoint of an intimate link between invariance and causation, while the proposed method directly explores conditional invariance given the proposed latent causal model.
Sun et al. (2021) mainly focus on single domain, while the proposed method consider multiple domains. The proposed method is also different from the work in [Liu et al.] (202T) in that the
former assume the latent content variable caused by the style variable, while the latter depends on a confounder to model the causal relation between the latent content variable and the style variable.
Unlike the work in Lu et al. (2021) that the label is treated as variable causing the other latent variables, the proposed method assumes that the label have no child nodes.
Causality for Domain Generalization It has been shown that there is closely the connection be- tween causality and generalization (Peters et al, 2016). Motivated by this, most of current methods leverage introduce new methods in various applications, e.g., domain generalization, (Mahajan jet al. 2021 Christiansen et al. 202 Wang et al. (002a) classification (Veitch et al.] 2021] Out-ot-Distributie Generalization Ahuja et al. (222)) Perhaps the closest our problem setting is domain generalization, where one can not "see' input data x. In general, because one can not 'see' input data x for domain generalization, obtaining identifiability result in the setting of domain gener- alization is generally not possible. In contrast, this work provides the identifiability result, providing a principled way to guarantee that the learned predictor can generalize to the target domain.
THE PROPOSED LATENT CAUSAL MODEL FOR LATENT COVARIATE SHIFT
We introduce a latent causal model to formulate features and label generative process to handle LCS as depicted by Figure 2 It introduces the observed domain variable D to denote in which specific domain data are collected. n. and n, represent some unmeasured factors corresponding to latent content noise and latent style noise, respectively. n. and n, influence the latent content variable z. and the latent style variable z,, respectively. Generally speaking, z, and z, should be dependent given the domain variable D. Here we consider that z, causes Z to model the correlation between 2, and y. In the proposed latent causal model, (z.) change across domains while Pr(ylee z,) is invariant across domains, which meets the basic assumption in the proposed latent covariate shift.
3

---

Under review as a conference paper at ICLR 2023
In the following, discuss two key causal assumptions, which also highlight the novelty of the proposed latent causal model.
Ze causes y: Previous works consider the causal relation between x and y as y x
Li et al., 2018). while we employ 2, y. We argue that these two cases are not contradictory since the labels in these two cases represent two different physical meanings. To understand this point, let ý replace in the first case (i.e.. y - x) to distinguish from y the second case (i.e., &e + y). For the first case, consider the following generative process of images. A label should be first sampled, e.g., y. then one may determine content information regarding to the label y, and finally generate a image, which is reasonable assumption in many real application scenarios. In the proposed latent causal model, n. play a role to replace ý and causes the content variable Ze- For the second case, Ze - y, formulates the process that experts extract content information from given images and then provide reasonable labels according to their domain knowledge. This assumption has been employed by some recent works (Mahajan et al., 2021] Liu al., [202]] Sun al. (202I). Particularly, these two different labels, y and y, has been simultancously considered in Mahajan et al. (2021).
{Gong et al., 2016 Stojanov et al.) 2019
Figure 2: The proposed latent causal model.
Ze causes z.: Here we consider having the object essence Ze first, from which a latent style z, springs to render x. The correlation between y and z, can be seen as a spurious correlation that should not contribute to predicting y. We here employ ze as a confounding factor of both y and z, to model the spurious correlation. The rationality of this assumption can be further verified from the viewpoint of the converse. In particular, if we assume that 2, causes Ze, all high-level information in input data 2, and z, would be causally related to the label y, which can not model the spurious correlation and is obviously unreasonable. Therefore, assuming Z, + Ze is more persuasive and consistent with previous works (Gong et al. 2016 Stojanov et al. 2019 Mahajan et al. 202 One recent work in [un et al. (2021) leverages an additional variable as a confounding factor that influences both the content variable z. and the style variable z, to model their relation. Interestingly, the identifiability result in Sun et al. (202I) does not depend on the confounding factor. As a result, the confounding factor can be incorporated into the domain index, which is equivalent to the case where 2e and z, are independent given the domain index. By contrast, the proposed latent causal model assumes a more general setting where ze and z, are dependent given the domain index, as depicted by Figure We will further verify the advantages of the proposed latent causal model in experiments, compared with Sun et al. (202T).
4 IDENTIFIABILITY ANALYSIS OF THE PROPOSED LATENT CAUSAL MODEL
In this section, we provide identifiability analysis for the proposed latent causal model. We first build a connection between the proposed latent causal model and nonlincar ICA by using the in- dependence among latent noise variables. We then show that it is still challenging to completely identify the proposed latent causal model due to transitivity, even with the identifiability result in nonlinear ICA. We finally show how to partially identify the latent content variable Ze, by integrating the identifiability result of nonlinear ICA (Khemakhem et al. 2020) with the dependence between n. and y.
4.1 RELATING THE PROPOSED LATENT CAUSAL MODEL WITH NONLINER ICA
The proposed latent causal model splits latent noise variables n into two disjoint parts, n. and
ng. as depicted by Figure 3 Since n, models the noise information, the latent noise variables
n. are assumed to be independent with each other in a causal system (Pearl 2000 Spirtes et al.) As result, it is natural to connect the latent noise variables n, with latent indepen-
2001
For convenience in the later parts, with slight abuse of definition, independent means that n; are mutually independent conditional the observed variable D.
4

---

Under review as a conference paper at ICLR 2023
dent variables in nonlinear ICA. Specifically, nonlinear ICA aims to separate independent latent variables conditional on D, e.g. N4, from observed mixing data, egg., x. generated by a nonlinear function. Recent progress in Khemakhem et al. (2020) shows that one recover ny up to per- mutation and scaling with relatively mild conditions, egg., there is an auxiliary observed variable, similar D in Figure which modulates the distributions of all independent latent variables n4- The identifiability result of n, holds in both source and target domains, since we have fea- ture data x from both source and target do- mains in the training phase for domain adapta- tion. However, the permutation indeterminacy implies that we can not determine which re- covered variables n4 correspond to n. (or ng) without further information. Therefore, using the identifiability result of nonlinear ICA only is insufficient to identify n. and n,, We will further discuss how handle the permutation indeterminacy in Section 4.3 Before that, we first analyze identifiability of the proposed la- tent causal model to provide deep insight.
F
n
Figure 3: Relating with Nonliner ICA.
4.2 COMPLETE IDENTIFIABILITY, THE NON-IDENTIRABILITY RESULT
Even with the identifiability result of nonlinear ICA, it is still challenging to completely iden- tify the proposed latent causal model. In partic- ular, we have the following result:
Proposition 4.1. With the identifiability result of nonlinear ICA under certain assumptions the proposed latent causal model is still uniden- tifiable without additional assumptions, due to transitivity in latent space.
Figure 4: Two equivalent graph structures.
Intuitively, the non-identifiability result above is because can not determine which path is the correct path corresponding to the net effect of na on x; e.g., n. + Ze + x and + + + x are equivalent causal structures in Figure This indeterminacy due to transitiviry, because if z. + z, + x. then z. + x can also be an alternative graph structure to generate same observed data, without further assumptions. For example, for simplicity, let us only consider one-dimensional 2. and 2.. According to the graph structure shown in the right column of Figure 4 assume that 2, == ne. 2, := + and x := f(21, 22) + E (case I). We then consider the graph structure shown in the left columi of Figure 4 assume that 2. == nes F := The and x = o g(=e, =.) + where g(=e, =.) [že, 2 =a] (case 2). Interestingly, we find that the causal models in case 1 and case 2 generate the same observed data x, which implics that there are two different causal models to interpret the same observed data, even with the identifiable n. and n,.- It often appears in latent causal discovery and seriously hinders the identifiability of latent causal models. For reader who may be interested in that problem, recommend recent work by Adams et al. (002))
4.3 PARTIAL IDENTIFIABILITY IDENTIFYING Z. UP SCALING
Although completely identifying the proposed latent causal model is challenging, for domain adap- tation application, we are only interested in the identifiability of ze, instead of the latent style variable 2,, since label y is only caused by z.. Thanks to the observed y from source domains, we have the following identifiability result:
Proposition 4.2. Assume that
(i) all latent noise variables n, can be identified up to permutation and scaling,
See APPENDIX A.3 for detailed assumptions.
5

---

Under review as a conference paper at ICLR 2023
the latent content variable ze in the proposed latent causal model can be identifiable up to scaling by using the dependence between n. and y" from source domains.
Condition (i) can be obtained from the identifiability result of nonlinear ICA (Khemakhem al. 2020). The proposition shows that the content variable can be identified up to scaling by using the identifiability result of nonlinear ICA and the dependence between y and ne. in which the iden- tifiability result of nonlinear ICA ensures to recover n; up to permutation and scaling, while the dependence removes the permutation indeterminacy identify n, and thus Ze. The scaling indeter- minacy of 2 results from the scaling indeterminacy of n, and the unknown mapping form n. to ee- The scaling indeterminacy of &e has no significance and can be ignored in latent space, since it can be 'absorbed' by the nonlinear mapping from 2e to y. For example, consider the recovered variable ae and its scaling scaling( &c). When we try to learn a invariant predictor g(.) form &. to y, the scaling indeterminacy can be 'absorbed' by a composition predictor, e.g.. g(scaling())).
LEARNING INVARIANT Po (y(z/z)) FOR MSDA
The identifiable 2. provides principled way to guarantee that we can learn the conditional distri- bution Pa (y|z.) that is invariant across domains and thus can be generalized to the target domain. Furthermore, since the identifiable n, is the only parent node of ze. learning Po (y(y/z,) can be trans- ferred into learning Po (y n2), which is also invariant across domains as depicted by Figure 2 In this section, we propose novel method show how to learn the invariant conditional distribution (y n.) for MSDA.
5.1 THE PROPOSED METHOO FOR LEARNING INVARIANT Pa Po(y/nn)
As analyzed in Section 4.3 the identifiability of z. is based on the identifiability result of nonlinear ICA, so we need to identify n, first. To meet the conditions of identifiable Khemakhem et al. (2020), we employ the following Gaussian prior on n. and n,:
mentioned in
p(n D) = p(n. D)p(n,|D) = (u (D). En, (H))) En,,,,
(1)
where 4a and Σ denote the mean and variance, respectively. Both depend on the domain variable D and can be implemented with multi-layer perceptrons. Since n, are to independent noise variables, > here a diagonal matrix. Some other exponential distributions, e.g., Laplace distribution, also meet the conditions of identifiable n, and thus also feasible (Khemakhem et al. 2020). We here use the Gassian prior since it is easy to leverage the reparameterization trick (Kingma & Welling 2013). The proposed Gaussian prior equation [gives rise the following variational posterior:
<nn x) q(n.) D, (), D, x) = NN (D,x), E',, (D, x)) (D,x), xD,x));
where μı' and V denote the mean and variance of the posterior, respectively; both of them depend on the domain variable D and the observed x and can be implemented by multi-layer perceptrons. Combining this with the Gaussian prior in equation] we can derive the following evidence lower bound (ELBO):
max Ey(n D.x) (p(x Dxc(v(n/D,x)lll(n)))
where Dxc denotes the Kulbback-Leibler divergence.
By maximizing the ELBO equation 3
then recover n, up to scaling and permutation. To remove the permutation indeterminacy as mentioned in Section 4.3 we can simultancously maxi- mize the correlation between yd and niς identify ng- Here employ the mutual information to maximize the dependence. As a result, we arrive at:
max
E Eggin D. ()(x -Die(q(n (xxe((( + T(n y
ELBO
where ((n S denotes the mutual information between of and S in source domains, and À is regularization hyper-parameter parameter that balances the ELBO and the mutual information (MI). The pro- posed method is termed iLCC-MSDA (identifiable Latent Causal Content for MSDA), including two components: ELBO and mutual information. The ELBO component ensures that n, can be recov- ered up scaling and permutation. The MI component handles the permutation, and thus ensures

---

Under review as a conference paper at ICLR 2023
which recovered n, corresponds to the latent content variables ne. In the implementation, the variational low bounder of mutual information proposed by Alemi et al. (2016) to approximate the mutual information in equation 4
max A Dx) (p(x D)) -
+ (D) (o(yr(n)))
(5)
A graphical depiction of the proposed il.CC-MSDA is shown in Figure
ELBO
Prior
p0a D)
Dasa l
Transfer
Clasofar
Encoder
Decoder
Figure 5: The proposed iLCC-MSDA to learn the invariant p(y[nz) for multiple source domain adaptation. C denotes concatenation, and S denotes sampling from the posterior distributions.
Constraining the independence among As we discussed, the performance of the proposed iLCC-MSDA above relies on assumptions on the identifiability n,, which requires that there are enough domains across which n, changes in order to well capture the statistical independence. However, in real applications, we may not have sufficient domains. To mitigate this issue, motivated by disentangled representations (Higgins et al.] 2017 Kim & Mnih 20018; Chen et al. 2018), we proposes to use a hyperparameter to enhance the independence among 74:-
Entropy regularization In the loss function in equation 5 we maximize the causal influence be- tween yd and n in source domains with mutual information. To encourage such causal influence in target domain, we can also maximize the mutual information between y and n? by minimizing the following conditional entropy:
Lent = -E (( -P log log)(
(6)
where 3d denotes the estimated label in the target domain. This regularization has been empirically used make label predictions more deterministic in previous works (Wang et al., 2020 Li al. 2021. while we consider it from the view of causality.
Therefore, our final loss function is:
max A(E (p(x D)) - 3Dxe (0 (g(n) (x)p)),)))l(m) + Exm.,IDx) (n(y((n))) + +ybents (7)
where B, A, y paramterrameterss that trade off the independence of n. and n,, the classifier and the entropy regularization loss terms.
EXPERIMENTS
6.1 EXPERIMENTS ON SYNTIETIC DATA
Dataset We conduct experiments on synthetic data, generated by the following process: we divide the latent variables into 5 segments, which are corresponding 5 domains. Each segment includes 1000 examples. Within each segment, we first sample the mean and the variance from uniform distributions [1, 2] and [0.3, 1] for the latent exogenous variables n, and n. respectively. Then for each segment, we generate Zg, X and y according to the following structural causal model:
2
2 2
MTP((ze: k.);
where following (Khemakhem et al. 2020 we mix the latent z, and 2, using a multi-layer percep- tron generate

---

Under review as a conference paper at ICLR 2023
the first 4 segments source domains, and the last segment as target domain. Figure shawssowws the true and recovered distributions of the exogenous variables ne. Due to the support of nonlinear ICA, the proposed iLCC-MSDA obtain the mean correlation coefficient (MCC) 0.96 hetween the original n, and the recovered. Due to the invariant conditional distribution ply|n even with the change of distribution of the exogenous variables n. as shown in Figure 6(a). the learned p(y ne) can generalize to target segment in principle way as depicted by the Figure 6(b) Due to the limited space, Figure 6(b)onll shows 200 samples for the true and predicted y.
Results In implementation,
W My MMM Mogh MiMM Mow
Recovered n.
(b) Predicted y on the target segment
Figure 6: The Result on Synthetic Data.
6.2 EXPERIMENTS ON REAL DATA
Dataset We further evaluate the proposed iLCC-MSDA on benchmark domain adaptation dataset PACS dataset (Li et al. p and Terra Incognita (Beery et al.) 2018). In the original PACS, the label distributions for any two domains is very similar (iee. Dxc R 0.1). This data is suitable for domain adaptation with conditional shift as shown Figure (b) where the label distribution remains unchanged, while it is not appropriate for the proposed setting where the label distribution
changes across domains, as shown in Figure (c). Therefore, we randomly sample the original PACS dataset to provide new PACS dataset where the label distribution changes, to generate three datasets, PACS (Dxc = 0.3), PACS (Dxc = 0.5) and PACS (Dxz = 0.7). Here Dxc = 0.3(0.5, 0.7) denotes that KL divergence of label distributions in any two different domains is approximately 0.3 (0.5, 0.7). See APPENDIX for details of label distributions.
Baselines We compare the proposed method with state-of-the-ar methods to verify its effective- ness. Particularly, we compare the proposed methods with empirical risk minimization (ERM), MCDA (Saito et al. 2018). M3DA (Peng et al., 2019)), LIC-MSDA (Wang et al. 2020). T-SVDNet (Li et al. 2021). IRM (Arjovsky et al. 2019). IWCDAN (Tachet des Combes et al. 2020) and
LaCIM (Sun al., 2021). In these methods, MCDA. M3DA, LIC-MSDA and T-SVDNet learn invariant representations for MSDA, while IRM, IWCDAN and LaCIM learn invariant conditional distributions, allowing label distribution to change. Details of implementation, including network architectures and hyper-parameter setting, are in the APPENDIX. All the proposed methods are averaged over 3 runs with standard deviation.
Ablation studies The bottom of Table T and presents the results for ablation studies. We can observe that entropy regularization equation 6 significantly increases the performance (around 10% and 5%) of the proposed method on both dataset. This justifies the importance of the causal relation between y and n., which is consistent with our model assumption. Besides, the hyper-paramete B also boosts the performance by enforcing the independence among the latent variables ni.
Results The results by different methods on PACS are presented in Table I We can observe that as the increase of KL divergence of label distribution, the performance of MCDA, M3DA, LIC-MSDA and T-SVDNet, which are based on learning an invariant representations, gradually degenerates. When the KL. divergence is about 0.7, the performance of these methods is worse than traditional ERM. Compared with IRM, IWCDAN and LaCIM, which allows label distribution to change across domains, the proposed iLCC-MSDA obtains the best performance, due to our theoretical supports. Tabiee2]epicts the results by different methods on challenging Terra Incognit. The proposed iLCC- MSDA achieves a significant performance gain on the challenging task LL7 Compared with the other methods, the proposed iLCC-MSDA is the only that is superior to ERM.
7 CONCLUSION
The key for domain adaptation is to understand how the joint distribution of features and label changes across domains. Previous works usually assume covariate shift conditional shift inter-
8

---

Under review as a conference paper at ICLR 2023
Table Classification results and ablation study on PACS data.
PACS (Drc 0.3)
Methods
Accuracy
-Cartoon
-Photo
-Sketch
Average
ERM
82.3 0.3
81.3 0.9
94,9 1: 0.2
76.2 1 0.7
83.6
MCDA Safto
0.6
85.1
96.6
70.1
82.1
M3SDA (Peny
79.6 +
86.6 +
97.1 0.3
83.3 +
86.6
LIC-MSDA Wang
2020
82.7 -2.41.33
84.9
96,9
75.3
84.9
T-SVDNet (Li
2021
81.8 1188100 0.3
86.5 16.2
9.2
80.7
1 0.8
86.3
IRM (Arjovsky el al. 2019
79.6 0.7
77.0
0.2
71.7
1 2.3
80.7
IWCDAN (Tacher des Combes et al. 2020
84.0 0.5
78.1 0.7
96.0 0.1
75.5
1.9
83.4
LaCIM (Sun et 2021
2021
63.1
72.6 1.0
82.7 1
71.5
0.9
72.5
(LCC-M)DA(OUrS)
81.1
95.9
86.0
87.4
PACS (Dxe 0.5)
ERM
85,41 0.6
76.4 0.5
94,4 0.4
85.0 0.6
85.3
MCDA Safto 2018
0.1
9..
84.1
84.0
M3SDA Peng
81.2
7
9.5
+
LIC-MSDA Wang
2020
1.5
2.62
0.6
8..1
T-SVDNet (L
2021
77.6 176110 1.7
9.22
86.4
1 0.2
85.6
IRM (Arjovsky el al
2019
0.3
711113
9.1
78.7
0.7
81.4
IWCDAN Tacher des Combes et al. 2020
79.2 1.6
0.7
95.6
82.1
2.2
82.4
LaCIM (Sun et al 2021
67.4 1.6
66.6 0.6
81.0 1.2
82.3
0.6
74.3
77.6
95.0 0.3
PACS (Dxe 0.7)
ERM
86. 0.6
76.8 0.3
94.6 1 0.4
81.3 2.0
84.7
MCDA Saito
80.8
74.1
94
77.9
81.8
M3SDA (Png
2019
82.7
76.2
1.0
0.7
80.8
83.6
LIC-MSDA Wang
2020
1.6
74.6 1.4
9.7
80.8
83.5
T-SVDNet (L
2021
8.8
74.7
0.3
74.5
33
81.9
IRM (Arjovsky el al. 2019
2019
8.
73.3 1.8
04.3
69.4
4.6
80.3
IWCDAN [acher des Combes et al. 2020
76.3 0.8
73.9 + 1.6
93.1 0.5
77.6
3.8
80.2
LaCIM (Sun el al. 2021
63.6 0.9
68.7
1.4
77.5 1 3.8
77.8
2.2
71.9
(LCC-MSDA(Curs)
90.7 0.3
0.72
95.8
83.0 =
86.0
LLCC-MSDA(OurS with 3
90.2 0.5
73.4 0.8
95.7
13.7200 0.4
82.7 = 0.7
85.5
(LCC-MSDA(OurS)
1.5
1.6
92.0 0.5
59.6
75.7
Table 2. Classification results on Terratncggntt.
Methods
Accuracy
+1228
++.44
+1.46
+L7
Average
ERM
54.1 + 2.8
62.3 - 0.7
44.7 + 0.9
74,5 + 2.6
58.9
MCDA ( Saito et al.
2018
4.19
6.2
42.7 - 0.3
64,8
1.8888 8.1
55.9
M3SDA Peng al. 2019
62.3+11
6.47
41.3
5..4
55.9
LIC-MSDA Wang
2020
5.7
1.3
45.7
69.1
55.3
T-SVDNet (Li et 202
58.2
0.3
45.6
68.2
58.5
IRM (Arjovsky et al.
2019
1.7
0.3
4.64
7..1
58.7
IWCDAN [Tachel des Combes al. 2020
$8.1
$9.3
43.8
$8.9 1.9
55.0
LaCIM (Sun et al. 2021
5.3
59,8 1.6
46.3
70.8 1.0
58.8
ILCC-MSDA(OOrs)
64.3
63.1
4..7
80.8
0.4
63.2
LLCC-MSDA(Ous)) with 3
56.3
61.5
0.7
45.2 0.3
80.1
0.6
60.8
(LCC-M)DAA(Uun) with
54.8 + 1.4
58.9 = 1.8
46.8 + 1.4
73.1 0.6
58.4
pret the change of the joint distribution, which may be restricted in some real applications. Hence, this work considers new and milder assumption, latent covariate shift. Specifically, we propose a latent causal model to precisely formulate the generative process of input features and label. We show that the latent content variable in the proposed latent causal model can be identified up to scaling. This inspires causal variable, resulting principled way to guarantee generalization target domains. Exper- iments demonstrate the theoretical results and the efficacy of the proposed method, compared with state-of-te ohethh--rtttttt methods across various data sets.
method to learn the invariant label distribution conditional on the latent

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the con-gaussian and heterogeneous cases. Advances Neural Information Processing Systems, 34, 2021.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, -ha-iChittopphe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, $4:3438-3455 2021.
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pas Invariant risk minimization. arXiv preprini arXiv: 0.00000..9,..... 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning, pp. 81-88, 2007.
Victor Bouvier, Philippe Very, Céline Hudelot, and Clément Chastagnol. Hidden covariate shift: A minimal assumption for domain adaptation. arXiv preprint arXiv:1907.12299 2019.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework for distribution generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 202.1.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural net- works. The journal of machine learning research, ((1))2096-2033) 2016.
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transac- tions on pattern analysis and machine intelligence,
2016.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao. Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International con- ference machine learning, pp.
PMLR, 2016.
I. Higgins, Loic Matthey, A. Pal, Christopher Burgess, Xavier Glorot, M. Botvinick, Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Jayyan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correct- ing sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference on Artificial Intel- ligence and Statistics, pp. 2207-2217. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2666 2658. PMLR, 2018.
Diederik P Kingma and Max Welling, arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning. pp- 5637-5664. PMLR, 2021.
Meelis Kull and Peter Flach. Patterns of dataset shift. In First International Workshop on Learning over Multiple Contexts (LMCE) at ECML-PKDDD, 2014.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550 2017.
Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1991-10000, 2021.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Liu. Learning causal semantic representation for -uttrf-disttibution prediction. Advances in Neural Information Processing Systems, 34, 2021.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of-disttibutiin generalization. International Conference on Learning Representutions, 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference Machine Learning, pp. 7313-7324. PMLR, 2021.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Boost- ing domain adaptation by discovering latent domains. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3771-3780, 2018
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1400-14115 2019.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7845):947-1012, 2016.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 3723-3732, 2018.
Bernhard Schölkopf. Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: TrX.v::206.647..7 2012.

---

Under review as a conference paper at ICLR 2023
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible- flow networks (gin). arXiv preprint arXiv: 00200000.4482.. 2020.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, Cambridge, MA, 2nd edition, 2001.
Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang. Data-driven approach to multiple-source domain adaptation. In The 22nd International Conference Artificial Intelli- gence and Statistics, pp. 3487-3496. PMLR, 2019.
Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations to learn? Advances in Neural Information Processing Systems, 34, 2021.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural information processing systems, 20, 2007.
Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tie- Yan Liu. Recov- ering latent causal factor for generalization distributional shifts. Advances in Neural Informa- tion Processing Systems, 34, 2021.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Informa- fion Processing Systems, 1927-11999,,,, 2020.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural net- works, 10(5):988-999, 1999.
Victor Veitch, Alexander D' Ammour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invari- ance to spurious correlations text classification. Advances in Neural Information Processing Systems, 34:16196 16208, 2021.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning combine: Knowledge aggregation for multi-source domain adaptation. In European Conference on Computer Vision, pp. 727-744. Springer, 2020.
Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-ff-distiibution generalization with causal invariant transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 375-385, 2022a.
Zengmao Wang. Chaoyang Zhou, Bo Du, and Fengxiang He. Self-paced supervision for multi- source domain adaptation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022b.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distribu- tions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pp. 631 -31. PMLR, 2014.
Luyu Yang, Yogesh Balaji, Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In European Conference on Computer Vision, pp- 608-624. Springer, 2020.
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pncumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(1
2018.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827.
PMLR, 2013.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Twet AAAI conference on artificial intelligence, 2015.
12

---

Under review as a conference paper at ICLR 2023
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gor- don. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp- 7523-7532. PMLR, 2019.
Sicheng Zhao, Bo Li, Pengfei Xu, Xiangyu Yue, Guiguang Ding, and Kurt Keutzer. Madan: multi- source adversarial domain aggregation network for domain adaptation. International Journal of Computer Vision, 129(8):2399-2424, 2021.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Data Details The commonly used datasets for multi-source domain adaptation, such as Digits- five, Office-Homee Home, PACS, and DomainNet, are not considered in this work, because for these dataset the label distributions of any two domains is very similar, which is suitable for domain adaptation with conditional shift as shown in Figure (b). However, these datasets the proposed setting where the label distribution changes across domains, as shown in Figure (c). Therefore, we resample the original PACS (Li et al. 2017) dataset, which contains 4 domains, Photo, Artpainting, Cartoon and Sketch, which shares the same seven categories. The KL divergence of label distributions of any two domains in the original PACS is very small, round 0.1. For obtain- ing dataset that meets the requirement of the proposed latent covariate shift, we filter the original dataset by re-sampling it, and obtain three new datasets with different the KL divergences of label distributions as depicted by Figure 7 The resampling process just randomly select some sample from the original PACS dataset, so that the labels distribution changes across domains. The labels distribution are depicted by Figure For Terra Incognita (Beery et al.] (018), the label distribution long-tailed at each domain, and each domain has a different label distribution, which is naturally applicable for setting. This work uses the four domains from the original data, L,28, L43, L46 and 1.7, which shares the same seven categories: bird, bobcat, empty, opossum, rabbit, raccoon, skunk, as depicted by Figure (Here 'L8' denotes the image data collected from the location 28.)
not appropriate for
PROVIOL-ESI
(a)
(b)
(c)
Figure Label distributions of the filtered PACS data.
aattccoggggo Vita
2000
1.54
0.53
1.6
(a)
Figure Label distributions of Terra Incognita data used in this work.
Implementation Details For the synthetic data, we used encoder, egg. 3-layer fully connected network with 30 hidden nodes for each layer, and decoder, e.g. 3-layer fully connected network with 30 hidden nodes for each layer. We use 3-layer fully connected network with 30 hidden nodes for prior model. Since this is ideal environment verify the proposed method, for hyper-paramedt we set B = and y = 0 to remove the heuristic constraints, and we set À = 1c - 2. For the real data, all methods used the same network backbone, ResNet-19 18 pre-traine me--triied on ImageNet. Since it can be challenging to train VAE on high-resolution images, we use extracted features by ResNet-18 as our VAE input. We then use l-yer fully connected networks the VAE encoder and decoder,
14

---

Under review as a conference paper at ICLR 2023
2- 2-layer fully connected network for the prior model, 2-layer fully connected network transfer we set [ 4, y = 0.1, le 4 for the proposed method on all
n. to Ze. For
datasets.
A.1 THE PROOF PROPOSITION
To prove non-identifiability. it is sufficient to show that several different graph structures can lead to the same observed data. particular, us consider the net effect of n. on x. There are two different paths to 'explain' the net effect of n. we have no limitation on the function class of edges, we can cut the path 2e + 2₂ off (e.g.. the left sub-figure of Figure 4 and obtain the same observed data depicted by the right sub-figure of Figure 4 Therefore, there is a causal graph depicted by the left sub-figure of Figure 4 are equivalent with the proposed latent causal model depicted by the right sub-figure of Figure
One path is n. Ze - In this case, since
A.2 THE PROOF PROPOSITION
As mentioned in Section 4.1 there the permutation indeterminacy and scaling indeterminacy in identifiable П;. The permutation indeterminacy implies that we are uncertain which the recovered variable î; are corresponding problem, since the parent node of 2, includes n. only, Ze can be identifiable up to scaling, i.e., 2e = (n,) where / can be any nonlinear function. Let consider the relationships between each n, and in the proposed causal model, it is clear that the label y depends on and is independent with n, given the domain variable D. As result, we can compute the correlations (e.g., by mutual information) between y from source domains and á from source domains to determine which recovered variables Ai are corresponding to the latent content variable n..
the latent content variable n.- If can solve this permutation
08-
28-
14-
14
20
0.
0.
(a)
(b)
63-
2.-
20
03 15 2.
03
10
(c)
(d)
Figure 9: The t-SNE visualizations of learned features n, different domains on the ~L77 task in Terralncognita. (a) The learned features n. in the L28 domain (b) The learned features n. in the L43 domain (c) The learned features n, the L46 domain (dyThe learned features n, the domain. We observe that the distribution of learned feature n. by the proposed method changes across domains, which is very different with the previous methods based on learning invariant rep- resentations.
15

---

Under review as a conference paper at ICLR 2023
A.3 ASSUMPTIONS FOR NONLINEAR ICA
Our theoretical results, e,g,, Propositions and rely on the identifiability result of nonlin- ICA. For completeness, we present assumptions for the identifiability result Khemakhem et al. Sorrenson et al. (2020). as follows:
2020
Theorem A.1. Khemakhem et al. 2020); Sorrenson al. 2020 Suppose the following conditional generative model:
n2 :~ (.
(9)
x: f(n) +e.
(10)
Assume the following holds:
(i) The set {x E (x)e(x) = 0} has measure zero (iee. has at most countable number of elements), where Se is the characteristic function of the density Pe-
(ii) The function f Eq. 10 is bijective.
(iii) There exist 2d + distinct points Dos di,..., D2a such that the matrix
L L= (Do), ... m(...) (())))
(11)
of size 2d 2d is invertible, where d is the dimension of n. (DD) denotes the vector of their coefficients, which depends on 3i,1 and B1.
then the true latent variables are related the estimated latent variables by the following relationship:
P
where P denotes the permatation matrix with scaling, denotes constant vector.
Eq. g is enforcing Gaussian distributions on the latent noise variables Note that the assumptions
of nonlinear ICA [Kemakheem et al., 2020) on the noise could be broad exponential family distri- bution, e.g., Gaussian distributions, Laplace, Gamma distribution and on. This work consider
Gaussian distribution, mainly because we implement it in our experiment, as shown Eqs. 1 and
16


Reviews:
[{"review_id": "pVPpoTFFoG-", "reviewer": "Reviewer_wpga", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies multi-source domain adaptation. Instead of learning domain-invariant features to address the conditional shift, this paper assmes latent covariate shift and proposes latent causal model to formulate the data and label generating process. The identifiability of the proposed model is analyzed. By integrating the identifiability with the dependence between nc and y, this paper proposes a method to learn the invariant conditional distribution pD(y|zc). Experiments on both synthetic and real datasets are conducted.", "summary_of_the_review": "Interesting idea, limited technical contribution, insufficient experiments and analysis and just so-so presentation. ", "strength_and_weaknesses": "Strengths:\n1. The idea of considering latent covariate shift is interesting and the motivation is clearly explained with examples.\n2. By combinining difference components, the proposed method achieves superior performance as compared to the compared baselines.\n\n\nWeaknesses:\n1. The latent casual model is naive simply by combining existing disentanglement and casual methods. The insights behind such combination are insufficient.\n2. The technical contribution is limited. The different components of the method, such as Gaussian prior, the mutual information and its approximation, entropy regularization, independence enhancement, are all existing techniques.\n3. The commonly used datasets for multi-source domain adaptation, such as Digits-five, Office-Home, and DomainNet, are not considered. Due to page limit, I expected to see the results in supplementary material.\n4. It is strange that when KL increases from 0.3 to 0.7, the performance of the baselines increase while the proposed method decreases. There is no analysis on the reasons.\nCommonly used visualizations for domain adaptation, such as GradCam, are missing. t-SNE is not convincing.\n5. The contributions of this paper are not well summarized.\n6. Some closely related recent methods on multi-source domain adaptaion are not introduced and compred, such as \"Self-paced Supervision for Multi-Source Domain Adaptation\", \"MADAN: multi-source adversarial domain aggregation network for domain adaptation\".\n7. The presentation needs to be improved. There are some simple grammar errors, such as \"a image\"->\"an image\", \"nc play a role\"->\"nc plays a role\". The full names of some abbreviations should be given for the first time use, such as ICA. The format of references, especially the names of conferences and journals, is inconsistent.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W6L1hCCUJtg", "reviewer": "Reviewer_fg87", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a domain adaptation setting where the features X and label Y are both influenced by latent features Z, the distribution of which may shift across domains. The idea of the paper is to fit a latent variable model to learn Z, and then use the learned variables as the basis for an invariant predictor. The bulk of the paper is concerned with showing this is indeed possible, which they aim to do by adapting results from non-linear independent component analysis.", "summary_of_the_review": "Although the core idea of applying non-linear identification to domain adaptation is interesting, this paper is not ready for publication.", "strength_and_weaknesses": "Strengths\n\nThis work fits in well with a recent line of attacks on domain adaptation that assume an underlying (causal) structure involving latent variables, then define the allowed class of shifts in terms of shifts on these latent variables. Such approaches seem promising for articulating more refined (and possibly even useful) domain shift assumptions. The idea of actually trying to learn the latent variables also seems sound, and relatively underexplored.\n\nWeaknesses\n\nThe main problem with this paper is that the main contribution is to give conditions for when the latent-variable modelling approach will work, but then these conditions get omitted!  Proposition 4.1 just references \"certain assumptions\"! The issue is that all of the promise of this approach comes down to whether or not these certain conditions are actually plausible in real problems. As such, one would expect the bulk of the paper to be dedicated to this question. Instead, the conditions aren't even given. \n\nA secondary---related---weakness is that the treatment of related work fails to make it clear how other approaches to domain adaption relate to the currently proposed one. This is most obvious with respect to latent variable based approaches---e.g., 1,2,3,4 at end of this review. But it's also important with respect to other baselines. For example, if Z_c can be perfectly reconstructed from X, then because Z_c is sufficient for Y under the assumed causal model, we would expect vanilla ERM to work for this problem! It's important to discuss how this, and other distributional approaches, might fail. \n\nFinally, there are some serious problems with clarity. As particular examples,\n1. it's unclear what the data setup here is. What qualifies as out of domain? What do we assume access to at training time? The setup seems to be non-standard, but it's not explicitly spelled out \n2. How exactly are n and z meant to differ? As latent variables, they are literally indistinguishable in the model in figure 2 (the nodes could be collapsed together without any observable implication)\n\n\nSome related work on domain adaptation with latent variables following a causal structure: \n1. https://proceedings.neurips.cc/paper/2021/hash/a8f12d9486cbcc2fe0cfc5352011ad35-Abstract.html\n2. https://arxiv.org/abs/2006.07500\n3. https://arxiv.org/abs/2106.00545\n4. https://arxiv.org/abs/2208.06987", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CULWwLje-p", "reviewer": "Reviewer_x2cf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors argue that existing works on domain adaptation are restricted to assuming covariate shift or conditional shift to interpret the change of the joint distribution and thus have their limitations in real applications. They propose a new latent casual model to formulate the generative process of input features and labels, they evaluated their proposed approach on synthetic and two real datasets. ", "summary_of_the_review": "My overall impression is that the current experiments and results section is somewhat weak in supporting the paper's claim. For example:\n\n1. The authors probably should consider adding evaluation metrics in the experiment section and why it is valid.\n\n2. The introduction for the two real datasets is not clear to me, e.g. the motivation to choose these datasets; what does L7, L28, L43, and L46 mean in Terra Incognita?; because the authors modified the original PACS dataset to perform the experiments will that introduce inductive bias for the evaluation? \n\n3. The Interpretation of the results is not convincing to me.  In Table 1, the authors argued that with the increase of KL divergence of label distribution, the performance of MCDA, M3DA(typo here, it should be M3SDA), LtC-MSDA and T-SVDNet gradually degenerates. However, in Table 1 we can see that MCDA and LtC-MSDA performed better when DKL=0.5 compared to DL=0.3 and then decreased under DKL=0.7 which is contradictory with the conclusions in the results. Secondly, there is only one sentence: \"due to our theoretical supports\", to support why iLCC-MSDA performs the best compared with IRM, IWCDAN and LaCIM.\n\n4. L7 appeared in the results section very suddenly, it was not mentioned earlier and I feel it is very difficult to follow the logic there.\n\nBesides, I also want to ask the authors how do you distinguish nc and ns, zc and zs in your model framwork? ", "strength_and_weaknesses": "This paper proposes to solve an important question in multi-source domain adaptation. Even though the latent content variable can only be identified together with the nonlinear ICA, the proposed framework shows the potential to solve this challenge. However, I think the current results section is weak, and the interpretation is not enough to demonstrate the superiority of this method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject