Subspace Identification for Multi-Source Domain Adaptation
Authors: Zijian Li
Ruichu Cai
Guangyi Chen
Boyang Sun
Zhifeng Hao
Kun Zhang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Domain Adaptation
Identification

Abstract: Multi-source domain adaptation (MSDA) methods aim to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Although current methods achieve target joint distribution identifiability by enforcing minimal changes across domains, they often necessitate stringent conditions, such as an adequate number of domains, monotonic transformation of latent variables, and invariant label distributions. These requirements are challenging to satisfy in real-world applications. To mitigate the need for these strict assumptions, we propose a subspace identification theory that guarantees the disentanglement of domain-invariant and domain-specific variables under less restrictive constraints regarding domain numbers and transformation properties and thereby facilitating domain adaptation by minimizing the impact of domain shifts on invariant variables. Based on this theory, we develop a Subspace Identification Guarantee (SIG) model that leverages variational inference. Furthermore, the SIG model incorporates class-aware conditional alignment to accommodate target shifts where label distributions change with the domain. Experimental results demonstrate that our SIG model outperforms existing MSDA techniques on various benchmark datasets, highlighting its effectiveness in real-world applications.

Subspace Identification for Multi-Source Domain Adaptation
Zijian Li Ruichu Cai²; Guangyi Chen Boyang Sun Zhifeng Hao', Kun Zhang 1 Carnegie Mellon University
2 School of Computer Science, Guangdong University of Technology Mohamed bin Zayed University of Artificial Intelligence Shantou University
Abstract
Multi-source domain adaptation (MSDA) methods aim to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Although cur- rent methods achieve target joint distribution identifiability by enforcing minimal changes across domains, they often necessitate stringent conditions, such as an adequate number of domains, monotonic transformation of latent variables, and invariant label distributions. These requirements are challenging to satisfy in real-world applications. To mitigate the need for these strict assumptions, we propose a subspace identification theory that guarantees the disentanglement domain-invariant and domain-speciinn variables under less restrictive constraints regarding domain numbers and transformation properties, thereby facilitating do- main adaptation by minimizing the impact of domain shifts invariant variables. Based on this theory. we develop Subspace Identification Guarantee (SIG) model that leverages variational inference. Furthermore, the SIG model incorporates class-aware conditional alignment to accommodate target shifts where label dis- tributions change with the domains. Experimental results demonstrate that our SIG model outperforms existing MSDA techniques on various benchmark datasets, highlighting its effectiveness in real- world applications.
Introduction
Multi-Source Domain Adaptation (MSDA) is method of transferring knowledge from multiple labeled source domains to an unlabeled target domain, to address the challenge of domain shift between the training data and the test environment. Mathematically, in the context of MSDA, we assume the existence of M source domains (S,, .... S.. } and a single target domain T. For each source domain S, data are drawn from a distinct distribution, represented as Px,y|us, where the variables x, y, correspond to features, labels, and domain indices, respectively. In similar manner, the distribution within the target domain is given by Px.y|uz. In the source domains, have access to m, annotated feature-label pairs each domain, denoted by (xx,,
(x, Siym, k= while in m the target domain, only my unannotated features are observed, represented as (x) (x7 k==1 The primary goal of MSDA is to effectively leverage these labeled source data and unlabeled target data to identify the target joint distribution Px.y|uy:
However, identifying the target joint distribution of x, y|u using only x|u7 as observations present significant challenge, since the possible mappings from Px,y Px|uy are infinite when extra constraints are given. To solve this problem, some assumptions have been proposed to constrain the domain shift, such covariate shift [44), target shift [58, 1]. and conditional shift [3, 56). For example, the most conventional covariate shift assumption posits that Py x is fixed across different domains
Conresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

while Px varies. Under this assumption, researchers can employ techniques such as importance reweighting [52], invariant representation learning [11]. or cycle consistency [17] for distribution alignment. Additionally, target shift assumes that Pxly is fixed while the label distribution Py changes, whereas conditional shift is represented by a fixed Py and a varying Pxly- More generally, the minimal change principle has been proposed, which not only unifies the aforementioned assumptions but also enables the theoretical guarantee of the identifiability of the target joint distribution. Specifically, assumes that Pylu and y.ly.u change independently and the change of Pxly. is minimal. Please refer to Appendix G for further discussion of related works, including domain adaptation and identification.
Although current methods demonstrate the identifiability of the target joint distribution through the minimal change principle, they often impose strict conditions on the data generation process and the number of domains, limiting their practical applicability. For instance, iMSDA [30] presents the component-we wise identification of the domain-changed latent variables, subsequently identifying target joint distribution by modeling a data generation process with variational inference. However, this identification requires the following conditions. First, sufficient number of auxiliary variables employed for the component-wise theoretical guarantees, meaning that when the dimension of latent variables is 71, a total of 2n + domains are needed. Second, in order to identify high-level invariant latent variables, component-wise monotonic function between latent variables must be assumed. Third, these methods implicitly assume that label distribution remains stable across domains, despite the prevalence of target shift in real-world scenarios. These conditions are often too restrictive to be met in practice, highlighting the need for a more general approach to identifying latent variables across a wider range of domain shifts.
In an effort to alleviate the need for such strict assumptions, we present a subspace identification and domain-speciinn
theory in this paper that guarantees the disentanglement of variables under more relaxed constraints concerning the number of domains and transformation properties. In contrast to component-wise identification, our subspace identification method demands fewer auxiliary variables (i.e., when the dimension of latent variables is n, only n + 1 domains are required). Additionally, we design a more general data generation process that accounts for target shift and does not necessitate monotonic transformation between latent variables. In this process, we categorize latent variables into four groups based on whether they are influenced by domain index label. Building on the theory and causal generation process, we develop a Subspace Identification Guarantee (SIG) model that employs variational inference identify latent variables. A class-aware condition alignment is incorporated to mitigate the impact of target shift, ensuring the update of the most confident cluster embedding. Our approach is validated through simulation experiment for subspace identification evaluation and four widely-used public domain adaptation benchmarks for application evaluation. The impressive performance that outperforms state-of-the-e methods demonstrates the effectiveness of our method.
2 Identifying Target Joint Distribution with Data Generation Process
2.1 Data Generation Process
We begin with introducing the data generation process. As shown in Figure 1, the observed data x X are generated by latent variables z E Z C R". Sequentially, we divide the latent variables z into the four parts, i.c. {Z1, Z2, 23, 21) Z C IR,", which are shown as follows.
u
y
Z
Z2
Z3
Z4
Z
domain-specific and label-irrelevant variables 2 E R" domain-specific but label-relevant variables Z2 E _2",, domain-invariant and label-relevant variables Z3 E RPs
x
domain-invariant but label-irrelevant variables Z4 E 1874.
To better understand these latent variables, we provide some examples in DomainNet datasets. First, Z₁ E "R denotes the styles of the images like "infograph" and "sketch",, which are irrelevant to labels. Z2 E RT22 denotes the latent variables that can be the texture information relevant to domains and labels. For example, the samples of "clocc"" and "telephone" contain some digits, and these digits these samples are special texture, which can be used for classification and be influenced by different
Figure 1: Data generation process, where the gray the white nodes de- note the observed and latent vari- ables, respectively.

---

styles, such as "infograph". Z3 I denotes the latent variables that are only relevant to the labels. For example, in the DomainNet dataset, it can be interpreted as the meaning of different classes like "Bicycle" or "Teapot". Finally, E R denotes the label-irrelevant latent variables. For example, 24 can be interpreted the background that is invariant to domains and labels.
Based on the definitions of these latent variables, we let the observed data be generated from through an invertible and smooth mixing function g : Z + X. Due to the target shift, we further consider that the Py is influenced by u, i.c. u y.
Compared with the existing data generation process like [30], the proposed data generation process different in three folds. First, Pu is independent of Py in the iMSDA [30], so the target shift is not taken into account. Second, the data generation process of iMSDA requires invertible and monotonic function between latent variables for component- priseeeettwiseeeee wise identification, which is too strict to be met in practice. Third, provide more general way to depict the real-world data, our data generation process introduces the domain-specific but label-relevant latent variables Zaz and the Nomain-invarianl but label-irreleaan variables 24.
2.2 Identifying the Target Joint Distribution
In this part, we show how to identify the target joint distribution Px,ylu with the help of marginal distribution. By introducing the latent variables and combining the proposed data generation process, we can obtain the following derivation.
Px.ylut
Pxy 22.22. d2, dZ2dz3d2.
22 Pylur dzzzdzzz
Pxla 22.2.1 Pe Py uy dzz dzzdzzdz,
According to the derivation Equation (1), we can identify the target joint distribution by modeling three distributions. First, need to model Px|2,,22.2,.4 implying that we need to model the conditional distribution of observed data give latent variables, which coincides with a generative model for observed data. Second, we need to estimate the label pseudo distribution of target domain Py ur- Third, we need to model Px, x,,,,,,, u, meaning that the latent variables should be identified with auxiliary variables u, under theoretical guarantees. In the next section, we will introduce how to identify these latent variables with subspace identification block-wiss wise identification results.
Subspace Identifiability for Latent Variables
In this section, we provide how to identify the latent variables in Figure 1. In detail, we first prove that Z2 is subspace identifiable and Zı, Z3 can be reconstructed from the estimated 21, 22, 23. Then we further prove that 24 block-wise identifiable.
u
Z
To clearly introduce the subspace identification theory, we employ a sim- ple data generation process [3] as shown in Figure 2. In this data gen- eration process, 2 E RT and ze E Rn denote the domain-specifie and Homaann-nvariann latent variables, respectively. For convenient, we let z = {z., zeJ. = + ne. Moreover, we assume z, = (2) and z, -2 (2)) And {u. x, denote the domain index, labels, and ob- served data, respectively. And we further let the observed data be generated from through an invertible and smooth mixing function g : 2 X. The subspace identification of z, means that for each ground- 2,,i, there exits
x
Figure 2: A sim- ple data generaliza- tion process for intro- ducing subspace iden- tification.
2, and an invertible function h; R" + R. such that 2,,i = h2(2,).
Theorem 1. (Subspace Identification of z..) We follow the data generation process in Figure 2 and make the following assumptions:
Al (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.c., Pz|u over Z and U.
3

---

A2 (Conditional independent): Conditioned on u, each is independent of any other 23 for i.j {I,... i the conditional distribution, i.e., qi : log Paclu-
i + j. i.e. log Pa) = E; 9i(2i, u) where q: qi(zi, u) is the log density of
A3 (Linear independence): For any E 2, S IR",, there exist n, + 1 values of u, i.e,, Uj with j = 0,1, Ra. such that these nt, vectors w(z, u₂) w(z, uo) with j = 1,, ng are linearly independent, where vector w(z, uz) is defined as follows:
(Qq1(zL,
(q, u)
Ogn, (2ne: ) )
w(2 11
(2)
021
8z:
Szn,
By modeling the aforementioned data generation process, 22 is subspace identifiable.
Proof sketch. First, we construct an invertible transformation h between the ground-truth z and estimated 2. Sequentially, we leverage the variance of different domains to construct full-rank linear system, where the only solution of de. is zero. Since the Jacobian of h is invertible, for each D2.
Rs,i, é {1,... n,), there exists a h2 such that 2,, (2) and 2g is subspace identifiable.
The proof can be found in Appendix B. The first two assumptions are standard in the component- wise identification of existing nonlinear ICA [30, 27]. The third Assumption means that Peju should vary sufficiently over ri + 1 domains. Compared to component-win identification, which necessitates 2n + domains and is likely challenging to fulfill, subspace identification can yield equívalent results terms of identifying the ground-trune latent variables with only n ++1 1 domains. Therefore, subspace identification benefits from more relaxed assumption.
Based on the theoretical results of subspace identification, we show that the ground-truth z Z2 and Z3 be reconstructed from the estimated 2 22 and 23. For case of exposition, we assume that Z 22, Zz. and 24 correspond to components in z with indices {1,... n11, {nr 1m1+1, +,....., + n2), (nz + n2
1. n2 + + ns), and {ni + 12 + ny + 1,.... ,n}, respectively.
Corollary 1.1. We, follow the data generation in Section 2. and make the following assumptions which are similar to Al-A3;
A4 (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e., Pz|u.y > 0 over Z, U, and y.
A5 (Conditional independent): Conditioned on u and y, each zi is independent of any other zj for i E (1,.... ,n), i * j. i.e. log Paluy (z y) = Σx ((22, y, where qi qi(zi, density of the conditional distribution, i.e., qs Po
U, y)
is the log
A6 (Linear independence): For any z E C RR" there exists nh + 122 + ni + combination of (u. y), i.e. j = 1.... U and c = 1, CC and U x C-1 ni + n2 + n., where and C denote the number of domains and the number of labels. such that these nı 12 + ng vectors w Uj - -w(z, Uo, Yo) are linearly independent, where w(z, Uj. e) is defined as follows:
w(z,11y7 (0e](21,u.)) u.y)
aqi u.y)
Oqn (2n, u. u.
(3)
0z:
Izn.
By modeling the data generation process in Section 2.1, 22 is subspace identifiable, and 2ı, 23 can be reconstructed from 21, 22 and 22, 23, respectively.
Proof sketch. The detailed proof can be found in Appendix B.2. First, we construct an invertible transformation h to bridge the relation between the ground-truth z and the estimated 2. Then, we repeatedly use Theorem three times by considering the chang- ing of labels and domains. Hence, we find that the values of some blocks of the Ja- cobian of h are zero. Finally, the Jacobian of h can be formalized as Equation (4). where J3 denotes the Jacobian of h and
Jdd oz: and i E (1,2,3, 4). Since 82,
J
1
J1. =
J =
h(-)) is invertible, Jn is a full-rank ma- trix. Therefore, for each 22,, i E {mi + 1. ..... nn + n2), there exists a 12, such that 22,1 hz(22). Moreover, for each 21,, E {1, ... ni + 1), there exists
2
2.3
J2.d =
Jn
(4)
J311
3,3
J3.
J 4.1
J
J 4.3
J

---

Label Predictor
ys
fp
X5
Pre-trained
Backbone
fr.fr
Bottleneck Networks
Z2
Pyluu
Decoder
fs.fr
X7
Networks
23
fo
Class-aware
Z4
Condition
Alignment
The framework of the Subspace Identification Guarantee model. The pre-trained backbone networks are used to extract the feature f from observed data. The bottleneck and far fo are used generate with reparameterization trick. Label predictor takes Z2, Zz, and as input to model Py u.2g Rt The decoder is used to model the marginal distribution. Finally, 22 is used for class-aware conditional alignment.
Figure 3:
h1s5 such that 21,; = (2,; 22). And for each 23,i. {m1 + +22 + 1, + + nz). there exists 13, such that 23,i h3. 23). Then we prove that 24 block-wise identifiable, which means that there exists an invertible function ha, s.t.24 ha
Lemma 2. [30] Following the data generation process in Section 2. and the assumptions A4-A6 in Theorem we further make the following assumption:
A7 (Domain Variability: For any set A₂ C Z) with the following two properties: 1) Ag has nonzero
probability measure, i.e. Piz E A {u u',y y'Hl 1 > for any u' E u and y' E e. 2) Az cannot be expressed as B2, Z1 Z2 Zz for any Bzr C 24.
Bui, U2 E U and y2 V. such that See A Pelui y dz + SaeA, Pzlue ys By modeling the data generation process in Section 2.1, the 24 is block-wise identifiable
The proof can found in Appendix B.3. Lemma shows that 24 can be block-wise identifiable when the Pzju changes sufficiently across domains.
In summary, can obtain the estimated latent variables with the help of subspace identification and block-wise identification.
4 Subspace Identification Guarantee Model
Based on the theoretical results, we proposed the Subspace Identification Guaranteed model (SIG) as shown in Figure 3, which contains a pariational-infeeenne--beee neural architecture model the marginal distribution and class-aware conditional alignment to mitigate the target shift.
4.1
Neural Architecture
According to the data generation process in Figure 1, we first derive the evidence lower bound (ELBO) in Equation (5).
ELBO -B
(x)x)) (xslx) In Px (x/z) + [ (xlx) In Pyluaazee (ylu, | 22, 23)
(5)
+ Eg (xix) In Puiz (u|z) Dx (xxx
Since the reconstruction of is not the optimization goal, remove the reconstruction of and rewrite Equation (5) as the objective function in Equation (6).
Luto Lvas + Ly
Lone Egg (xlx)) In (x(2))())z) + DxL (xr((2xxx(/xx)]xooom
(6)
C, -E Engis (4/x)) In Pylu.as (y [1 22, zz).
To minimize the pairwise class confusion, we further employ the minimum class confusion [25] into the classification loss Cy. According to the objective function in Equation (6), illustrate how to
5

---

implement the SIG model in Figure 3. First, we take the observed data and X from the source domains and the target domain as the imputs of the pre-trained backbone networks like ResNet50 and extract the feature Js, and JT. Sequentially, we employ bottleneck networks, Jn and fa. to extract the latent variables z. Then, we take z to reconstruct the pre-trained features via an MI.P-based decoder to estimate the marginal distribution Px|u- Finally, we take the 22, 23, and the domain embedding to predict the source label to estimate Py 1.22,.85
encoder, which contains
4.2 Class-aware Conditional Alignment
To estimate the target label distribution Pyluz and mitigate the influence of target shift, we propose the class-aware conditional alignment automatically adjust the conditional alignment loss for each sample. Formally, can be written as
La = =2
1 Σ (0)
Poro 123.5
((()) 23,7112. w = exp (- (pH(p))))),
where C' denotes the class number; 3.. and T. denote the latent variables of ith class from source and target domain, respectively; w denotes the prediction uncertainty of each class in the target domain; Py(s) denotes the estimated label probability density of ith class; H denotes the entropy.
The aforementioned class-aware conditional alignment based on the existing conditional alignment loss, which can be formalized in Equation (8).
La =
1 C
Σ
2 T,2,,
(8)
3.5
However, conventional conditional alignment usually suffers from two drawbacks including misesti- mated centroid and low-quality pseudo-labels. First, the conditional alignment method implicitly assumes that the feature centroids from different domains are the same. But it is hard to estimate the correct centroids of the target domain for the class with low probability density. Second, conditional aligmment heavily relies on the quality of the pseudo label. But existing methods usually use pseudo labels without any discrimination, which might result in false alignment. To solve these problems, we consider two types of reweighting.
Distribution-based Reweighting for Misestimated Centroid: Although the conditional alignment method implicitly assumes that the feature centroids from different domains are the same, it is hard to estimate the correct centroids of the target domain for the class with low probability density. To address this challenge, one straightforward solution to consider domain. To achieve this, employ the technique of black box shift estimation method (BBSE) [37] to estimate the label distribution from the target domain py. So we use the estimated label distribution to reweight the conditional alignment loss in Equation (8).
label distribution of the target
Entropy-based Reweighting for Low-quality Pseudo-labels: conditional alignment heavily relies on the quality of the pseudo label. However, existing methods usually use pseudo labels without any diserimination, which might result in false alignment. To address this challenge, we consider the prediction uncertainty of each class in the target domain. Technologically, for each sample in the target dataset, we calculate the entropy-based weights via the prediction results which are shown as w()) in Equation (7).
By combining the distribution-baat weights and the entropy-based weight, we can obtain the class-aware conditional alignment shown in Equation (7). Hence the total loss of the Subspace Identification Guarantee model can be formalized as follows:
Liotal = Ly + BLace + aLaa
(9)
where o;, B denote the
Experiments
5.1 Experiments on Simulation Data
In this subsection, we illustrate the experiment results of simulation data to evaluate the theoretical results of subspace identification in practice.

---

5.1.1 Experimental Setup
Data Generation. We generate the simulation data for binary classification with 8 domains. To better evaluate our theoretical results, we follow the data generation process in Figure 2, which includes two types of latent variables, i.e., domain-specifie latent variables z, and domain-invarian latent variables Zr- We let the dimensions of 2, and z, be both 2. Moreover, 2, are sampled from w different mixture of Gaussians, and že are sampled from factorized Gaussian distribution. We let the data generation process from latent variables to observed variables be MLPs with the Tanh activation function. We further split the simulation dataset into the training set, validation set, and test set.
Evaluation Metrics. First, we employ the accuracy of the target domain data to measure the classification performance of the model. Second, we compute the Mean Correlation Coefficient (MCC) between the ground-truth 25 and the estimated & on the test dataset to evaluate the component- wise identifiability of comain--ppccffe latent variables. A higher MCC denotes the better identification performance the model can achieve. Third, to evaluate the performance of subspace identifiability of domain-specific latent variables, first use the estimated Σ. from the validation set regress each dimension of the ground-truth 2s from the validation set with the help of a MLPs. Sequentially, we take the &. from the test set as input estimate how well the MLPs can reconstruct the ground- 2, from the test set, so we employ Root Mean Square Error (RMSE) to measure the extent of subspace identification. A low RMSE denotes that there exists a transformation h; between že,a and 2,, & 2. i.e z,,i h4 2). i E (1, 2}. For the scenario where the number of domains is less than 8, we first fix the target domain and then try all the combinations of the source domains. And we publish the average performance of all the combinations. We repeat each experiment over 3 random seeds.
5.1.2 Results and Discussion
The experimen- tal results of the simulation dataset are shown in Table 1. According to the exper- iment results, we can obtain
Table 1: Experiments results on simulation data.
State
U
ACC
MCC
RMSE
8
0.9982(0.0004
0.9037(0.0087)
0.0433(0.005))
Component-wiss Identification
0.9982(0.0007) 6
0.8976(0.0162)
0.0439(0.0073)
5
0.9982(0.0007)
0.8973(0.013))
0.0441(0.0055)
Subspace Identification
4
0.9233(0. 2039)
0.8484(0. 1452)
0.0582(0.0431)
3
0.8679(0.2610)
0.8077(0.1700) 1709)
0.0669(0.0482)
the following conclusions: 1) We can find
No Identification
2
0.5978(0.3039)
0.6184(0.2093)
0. 0.1272(0.0608)
that the values of MCC increase along with the number of domains. Moreover, the values of MCC high (around 0.9) and stable when the number of domains is larger than 5. This result corresponds to the theoretical result of component identification, where a certain number of domains (i.e. identification. 2) We can find that the values of RMSE
2n + 1) necessary for component decrease along with the number of domains. Furthermore, the values of RMSE are low and stable (less than 0.07) when the number of domains is larger than 3, but it drops sharply when These experimental results coincide with the theoretical results of subspace identification as well as the intuition where a certain number of domains are necessary for subspace identification (i.e. n. + 1). 3) According to the experimental results of ACC, we can find that the accuracy grows along with the number of domains and its changing pattern is relevant to that of RMSE, i.e., the performance is stable when the number of domains is larger than 3. The ACC results also indirectly support the results of subspace identification, since one straightforward understanding of subspace identification is that the domain-spedd information is preserved in &g. And the latent variables are well disentangled with the help of subspace identification, which benefits the model performance.
2.
5.2
Experiments on Real-world Data
5.2.1 Experimental Setup
Datasets: We consider four benchmarks: Office-Home, PACS, ImageCLEF, and DomainNet. For each dataset, we let each domain be target domain and the other domains be the source domains. For the DomainNet dataset, we equip cross-attention module to the ResNet101 backbone networks for
7

---

Table Classification results on the Office-Home and ImageCIEF datasets. For the Office-Homi dataset, We employ ResNet50 as the backbone network. For the ImageCLEF dataset, we employ R as the backbone network.
Office-Home
ImageCIEF
Model
Art
Clipart
Product
RealWorld
Average
P
C
I
Average
Source Only [16]
64.5
52.3
77.6
80.7
68.8
77.2
92.3
88.1
85.8
DANN [11]
64.2
58.0
76.4
78.8
69.3
77.9
93.7
91.8
87.8
DAN [40]
68.2
57.9
78.4
81.9
71.6
77.6
93.3
92.2
87.7
DCTN [69]
66.9
61.8
79.2
77.7
71.4
75.0
95.7
90.3
87.0
MFSAN [81]
72.1
62.0
80.3
81.8
74.1
79.1
95.4
93.6
89.4
WADN [54]
75.2
61.0
83.5
84.4
76.1
77.7
95.8
93.2
88.9
iMSDA [30]
75.4
61.4
83.5
84.4
76.2
79.2
96.3
94.3
90.0
SIG
76.4
63.9
85.4
85.8
77.8
79.3
97.3
94.3
90.3
better usage of domain knowledge. We also employ the aligmment of MDD [78]. For the Office-Home and ImageCLEF datasets, we employ the pre-trained ResNet50 with an MLP-based classifier. For the PACS dataset, we use R8 8 with an MI.P-based classifier. The implementation details are provided in the Appendix C. We report the average results over random seeds.
Baselines: Besides the classical ap- proaches for single source domain adaptation like DANN [11], DAN [40], MCD [50], and ADDA [61]. We also compare our method with several state-of-the-aa afrt multi-source domain adaptation methods, for ex- ample, MIAN-y [45], T-SVDNet
Table 3: Classification results on the PACS datasets. We employ ResNet18 as the backbone network. Experiment results of other compared methods are taken from ([30)).
Model
C
S
Average
Source Only [16]
74.9
72.1
94.5
64.7
76.7
DANN [11]
81.9
77.5
91.8
74.6
81.5
[33], LiC-MSDA [63], SPS [64]. and PFDA [10]. Moreover, we further consider the WADN [54], which is devised for the target shift of multi-source domain adaptation. For a fair comparison, we employ the same pre-train backbone net- works instead of the pre- pre-trained fea- tures for WADN in the original pa- per. We also consider the latest
MDAN [79]
79.1
76.0
91.4
72.0
79.6
WBN [43]
89.9
89.7
97.4
58.0
83.8
MCD [50]
88.7
88.9
96.4
73.9
87.0
M3SDA [46]
89.3
89.9
97.3
76.7
88.3
CMSS [70]
88.6
90.4
96.9
82.0
89.5
LIC-MSDA [63]
90.1
90.4
97.2
81.5
89.8
T-SVDNet [33]
90.4
90.6
98.5
85.4
91.2
iMSDA [30]
93.7
92.4
98.4
89.2
93.4
SIG
94.1
93.6
98.6
89.5
93.9
iMSDA [30], which addresses the MSDA via component wise identification.
5.2.2 Results and Discussion
Experimental results on Oifice-Home, ImageCLEF, PACS, and DomainNet are shown in Table 2, 3, and 4, respectively. Experiment results of other compared methods provided in Appendix D.2.
According the experiment results of the Office-Home dataset on the left side of Table 2, our SIG model significantly outperforms all other baselines on all the transfer tasks. Specifically, our method outperforms the most competitive baseline by a clear margin of 1.3% 4% and promotes classification accuracy substantially on the hard transfer task, e.g. Clipart. It is noted that our method achieves a better performance than that of WADN, which is designed for the target shift scenario. This is because of labels but also identifies the latent variables the data generation process. Moreover, our SIG method also outperforms iMSDA, indirectly reflecting that the proposed data generation process is closer to the real-worle setting and the subspace identification can achieve better disentanglement performance under limited auxiliary variables.
method not only considers how the domain variables influence the distribution
For datasets like ImageCLEF and PACS, our method also achieves the best-averaged results. In detail, we achieved comparable performance in all the transfer tasks in the ImageCI.FF dataset. In the PACS dataset, our SIG method still performs better than the latest compared methods like iMSDA and T-SVDNet in some challenging transfer tasks like Cartoon. Finally, also consider the

---

Table Classification results on the DomainNet datasets. We employ ResNet101 as the backbone network. Experiment results of other compared methods are taken from ([34) and [64]).
Model
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Average
Source Only [16]
52.1
23,4
47.6
13.0
60.7
46.5
40.6
ADDA [61]
47.5
11.4
36.7
14.7
49.1
33.5
32.2
MCD [50]
54.3
22.1
45.7
7.6
58.4
43.5
38.5
DANN [11]
60.6
25.8
50.4
7.7
62.0
51.7
43
DCTN [69]
48.6
23.5
48.8
7.2
53.5
47.3
38.2
SDA [46]
58.6
26.0
52.3
6.3
62.7
49.5
42.6
ML MSDA [35]
61.4
26.2
51.9
19.1
57.0
50.3
44.3
meta-MCD [32]
62.8
21.4
50.5
15.5
64.6
50.4
44.2
LIC-MSDA [63]
63.1
28.7
56.1
16.3
66.1
53.8
47.4
CMSS [70]
64.2
28.0
53.6
16.9
63.4
53.8
46.5
DRT+ST [34]
71.0
31.6
61.0
12.3
71.4
60.7
51.3
SPS [64]
70.8
24.6
55.2
19,4
67.5
57.6
49.2
PFDA [10]
64.5
29.2
57.6
17.2
67.2
55.1
48.5
iMSDA [30]
68.1
25.9
57.4
17.3
64.2
52.0
47.5
SIG
72.7
32.0
61.5
20.5
72.4
59.5
53.0
Art
Clipart
Product
RealWorld
Average
77
86
86
78
Accuracy
64
76
85
85
77
62
75
84
84
76
74
60
83
83
75
SIG-vae
SIG-sem
SIG
Figure 4: Ablation study on the Office-Home dataset. we explore the impact of different loss terms.
most challenging dataset, DomainNet, which contains more classes and more complex domain shifts. Results in Table 4 show the significant performance of the proposed SIG method, which provides 3.3% averaged promotion, although the performance in the task of Sketch is slightly lower than that of DRT+ST. Compared with iMSDA, our SIG overpasses by large margin under a more general data generation process.
To evaluate the effectiveness of individual loss terms, we also devise the two model variants. 1) SIG-sem: remove the class-aware alignment loss. 2) SIG-vae: remove the reconstruction loss and the KL divergence loss. Experiment results on the Office-Home dataset are shown in Figure 4. We can find that the class-aware alignment loss plays an important role in the model performance, reflecting that the class-aware alignment can mitigate the influence target shift. We also discover that incorporating the reconstruction and KI. divergence has positive impact on the overall performance of the model, which shows the necessity modeling the marginal distributions.
Ablation Study:
6 Conclusion
This paper presents a general data generation process for multi-source domain adaptation, which coincides with real-world scenarios. Based this data generation process, prove that the changing latent variables are subspace identifiable, which provides a novel solution for disentangled representa- tion. Compared with the existing methods, the proposed subspace identification theory requires fewer auxiliary variables and frees the model from the monotonic transformation of latent variables, making possible apply the proposed method real-world data. Experiment results several main- stream benchmark datasets further evaluate the effectiveness of the proposed subspace identification guaranteed model. In summary, this paper takes a meaningful step for causal representation learning. Broader Impacts: SIG disentangles the latent variables to create a model that is more transparent, thereby aiding the reduction of bias and the promotion of fairness. Limitation: However, proposed subspace identification still requires several assumptions that might not be met in real- weal-world scenarios. Therefore, how further to relax the assumptions, i.e., conditional independent assumption, would be interesting future direction.

---

7 Acknowledgements
We are very grateful to the anonymous reviewers for their help in improving the paper. This research supported in part by the National Key R&D Program of China (20212D011 the National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052), the major key project of PCL (PCL2021A12) This project is also partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract ROIHL 159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
References
[1] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint 1XXix:1903,0934 2019.
[2] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems, 29, 2016.
[3] Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentan- gled semantic representation for domain adaptation. In IJCAI: proceedings of the conference, volume 2019, page 2060. NIH Public Access, 2019.
[4] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain- specific batch normalization for unsupervised domain adaptation. Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pages 7354-7362 2019.
[5] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discrimi- native feature learning for unsupervised deep domain adaptation. In Proceedings the AAAI conference on artificial intelligence, volume 33, pages 3296-3303, 2019.
[6] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 627-636, 2019.
[7] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. dis- criminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pages 1081-1090. PMLR, 2019.
[8] Yuansi Chen and Peter Bühlmann. Domain adaptation under structural causal models. The Journal of Machine Learning Research, 22(1):11856-11 2021.
[9] Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287- 314, 1994.
[10] Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16654-16663,
2021.
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pages 1180-1189. PMLR, 2015.
[12] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
[13] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. International conference machine learning, pages 2839-2848 PMLR, 2016.
10

---

[14] Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series, In Conference on Uncertainty Artificial Intelligence, pages 939-948. PMLR, 2020.
[15] Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34:1624-1633, 2021.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Judy Hoffmann Eric Tzeng. Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. Pmlr, 2018
[18] Aapo Hyvärinen. Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1984)20110533) 2013.
[19] Aapo Hyvarinen, Juha Karhunen, and Erkki Oja. Independent component analysis. Studies in informatics and control, 11(2):005-277 2002.
[20] Aapo Hyvärimen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. arXiv preprint arXiv:2302.02672, 2023.
[21] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[22] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[23] Aapo Hyv"arinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
[24] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary vari- ables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[25] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 464-480. Springer, 2020.
[26] Guoliang Kang, Lu Jiang, Yunchao Wei, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for single-and multi-source domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 44(4):1793-180)) 2020.
[27] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-22111 PMLR, 2020.
[28] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Iden- tifiable conditional energy- based deep models based on nonlinear ica. Advances in Neural 2020.
Information Processing Systems,
[29] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10930, 2020.
[30] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pages 11455-11472. PMLR, 2022.
11

---

[31] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arX0vv1711.00848. 2017.
[32] Da Li and Timothy Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI. pages 382-403. Springer, 2020.
[33] Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9991-10000, 2021.
[34] Yunsheng Li, Lu Yuan, Yinpeng Chen, Pei Wang, and Nuno Vasconcelos. Dynamic transfer for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10998-11007, 2021.
[35] Zhenpeng Li, Zhen Zhao, Yuhong Guo, Haifeng Shen, and Jieping Ye. Mutual learning network for multi-source domain adaptation. arXiv preprint arXiv:2003. 12944, 2020.
[36] Zijian Li, Ruichu Cai, Tom ZJ Fu, and Kun Zhang. Transferable time-series forecasting under causal conditional shift. arXiv preprint
11.0222, 2021.
[37] Zachary Lipton, Yu- Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.
[38] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pages 41 14
4124. PMLR, 2019.
[39] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258, 2019.
[40] Mingsheng Long, Yue Cao, Jianmin Wang. and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, PMLR, 2015.
[41] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208-2217. PMLR, 2017.
[42] Sara Magliacane. Thijs Van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems, 31, 2018
[43] Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Boosting domain adaptation by discovering latent domains. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3771-3780, 2018.
[44] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22110)1345-1139, 2010.
[45] Geon Yeong Park and Sang Wan Lee. ieformatioonthhoreee regularization for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9214-9223, 2021.
[46] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-141 1415, 2019.
[47] Chuan-Xian Ren, Yong-Hui Liu, Xi-Wen Zhang, and Ke- Huang. Multi-source unsupervised domain adaptation via pseudo target domain. IEEE Transactions on Image Processing. 31:2122-
2 2135, 2022.
12

---

[48]
Manley Roberts, Pranav Mani, Saurabh Garg, and Zachary Lipton. Unsupervised learning under latent label shift. Advances in Neural Information Processing Systems, 35:18763-18778, 2022.
[49]
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210-20229, 2021.
[50] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018.
[51] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5);612-6) 634, 2021.
[52] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- log-likelhhood function. Journal of statistical planning and inference, 90(2):227 24,4 2000.
[53] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised 2018.
domain adaptation. arXiv preprint
[54] Changjian Shui, Zijian Li, Jiaqi Li, Christian Gagné, Charles X Ling, and Boyu Wang. Aggre- gating from multiple target-shifted sources. In International Conference on Machine Learning, pages 9638-9648. PMLR, 2021.
[55] Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang. Data-driven approach to multiplc-source domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3487-3496. PMLR, 2019.
[56] Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations to learn? Advances in Neural Information Processing Systems, 34:24791-24803, 2021.
[57] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[58] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33: 19276-9289, 2020.
[59] Takeshi Teshima, Issei Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mechanism transfer. In International Conference on Machine Learning, pages PMLR, 2020.
[60] Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference Machine Learning, pages 10401-10412. PMLR, 2021.
[61] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 7167-7176, 2017.
[62] Eric Tzeng. Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3477 141..347, 2014.
[63] Hang Wang. Minghao Xu. Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28 2020, Proceedings, Part VIII 16, pages 727-744. Springer, 2020.
[64] Zengmao Wang, Chaoyang Zhou, Bo Du, and Fengxiang He. Self-paced supervision for multi- source domain adaptation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022.
13

---

[65] Jun Wen, Nenggan Zheng, Junsong Yuan, Zhefeng Gong, and Changyou Chen. Bayesian uncertainty matching for unsupervised domain adaptation. arXiv preprint arXiv:1966.09692, 2019.
[66] Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi-source domain adaptation. In International Conference on Machine Learning, pages 10214-10224. PMLR, 2020.
[67] Shaoan Xie, Lingjing Kong, Mingming Gong, and Kun Zhang. Multi-domain image generation and translation with identifiability guarantees. In The Eleventh International Conference Learning Representations.
[68] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In International conference machine learning, pages 5432. PMLR, 2018.
[69] Ruijia Xu, Ziliiang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3964-3973, 2018.
[70] Luyu Yang, Yogesh Balaji. Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28. 2020, Proceedings, Part XIV 16, pages 608 624. Springer, 2020.
[71] Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. arXiv preprint arXiv:2210. 13647, 2022.
[72] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2 10.05428, 2021.
[73] Kun Zhang and Laiwan Chan. Kernel-based nonlinear independent component analysis. In Independent Component Analysis and Signal Separation: 7th International Conference, ICA 2007, London, UK, September 9-12. 2007. Proceedings 7, pages 301-308. Springer, 2007.
[74] Kun Zhang and Laiwan Chan. Minimal nonlinear distortion principle for nonlinear independent component analysis. Journal of Machine Learning Research, 9NNov):2455225 2008,
[75] Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
[76] Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, Qingsong Liu, and Clark Gly- mour. Domain adaptation as problem of inference on graphical models. Advances in neural information processing systems,
2020.
[77] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference machine learning, pages 819-827. PMLR, 2013.
[78] Yuchen Zhang. Tianle Liu, Mingsheng Long. and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International conference on machine learning, pages 7404-7413. PMLR, 2019.
[79] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.
[80] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica with uncon- ditional priors. ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
[81] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. In Proceedings the AAAI conference on artificial intelligence, volume 33, pages 5989-5996, 2019.
14

---

[82] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. JEEE transactions on neural networks and learning systems, 3114)1173- 2020.
15


Reviews:
[{"review_id": "kXhlJZqgBy", "reviewer": "Reviewer_Yx4K", "summary": "This paper proposes a novel framework to tackle the problem of multi-source unsupervised domain adaptation. Existing methods often have stringent requirements, such as a large number of domains and invariant label distributions. To address these limitations, this paper proposes a subspace identification theory that disentangles domain-invariant and domain-specific variables under less restrictive constraints. The proposed Subspace Identification Guarantee (SIG) model is based on variational inference and additionally incorporates class-aware conditional alignment to handle domain shifts. Experimental results demonstrate that the SIG model outperforms existing SOTA techniques on four benchmark datasets (OfficeHome, ImageCLEF, PACS, DomainNet), showcasing its effectiveness in real-world applications.", "questions": "Please see weaknesses. Also, the derivation of the ELBO in Equation 5 should be provided to enhance the completeness of the paper.", "limitations": "The authors discuss the limitations adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper defines three major drawbacks of [a]  – invariant label distribution, requiring a large number of domains and monotonic transformation between latent variables. These assumptions are relaxed by the *Subspace Identification Guarantee* (SIG) model proposed in this paper. Most interestingly, while Kong et al. claimed that for $n$ dimensional latent space, $2n+1$ domains would be necessary, authors here have shown that “n+1” domains are sufficient. \n- The core ideas on subspace identifiability are theoretically grounded. The paper defines a new data generative process based on a 4-way split of the latent space and builds upon the theoretical results to define a new multi-source unsupervised domain adaptation framework using a VAE-based architecture.\n- Results are presented on 4 benchmark datasets, with the proposed model outperforming all other SOTA approaches.\n\n[a] Kong et. al., Partial Identifiability for Domain Adaptation, ICML 2022", "weaknesses": "- In the data generation process, the paper introduces 4 latent variables from the product of two sets {domain-specific, domain-invariant} $\\times$ {label-relevant, label-irrelevant}. I am unsure what the intuitive reasoning behind the \"domain-invariant label-irrelevant\" latent variable $\\mathbf{z}_4$ is. \n- There are no individual constraints on $\\mathbf{z}_1$ and $\\mathbf{z}_4$ during the learning process. Are they identifiable individually?\n- While the paper claims that the proposed SIG model can handle all three types of shifts: covariate, target, and conditional, experiments focus primarily on covariate shift (based on the datasets used). It’s unclear how the datasets, which contain mostly covariate shifts, were used for studying the two other shifts.\n- The clarity could be improved, especially when introducing the SIG model. For example, in Section 4.2, $\\hat{\\mathbf{z}}_{3,\\mathcal{S}}^{(i)}$ is denoted as \"..latent variables of i-th class from source\". How is this estimated? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the valuable commands and helpful suggestions on our paper and the time dedicated to reviewing it. Below please see our point-to-point responses to your comments and suggestions.\n\n>**Q1**: I am unsure what is the intuitive reasoning behind the \"domain-invariant label-irrelevant\" latent variable $z_4$ is.\n\n**A1**: Thanks for your valuable questions which have improved the readability of our paper. We have provided the intuitive reasoning behind the domain-invariant and label-irrelevant latent variables $z_4$ and will include it in the final version. Although $z_4$ is not unrelated to domains or labels, it is an important component of the causal generation process. Moreover, reasoning $z_4$ with the block-wise identification guarantee can disentangle $z_4$ with other latent variables i.e., $z_1, z_2, z_3$, which makes it convenient to precisely identify other latent variables with subspace identification guarantee. \n\n>**Q2**: There are no individual constraints on $z_1$ and $z_4$  during the learning process. Are they identifiable individually?\n\n**A2**: We deeply value your careful reviews for the preciseness of our work. Actually, $z_1$ and $z_4$ are not identified individually. According to ELBO as the equation below, we can employ $\\textcolor{blue}{\\ln p_{{u}|{z}}({{u}|{z}})= \\ln p_{{u}|{z}} ({u}| z_1,z_2,z_3)}$ as a constraint on $z_1$. It indicates that $z_1$ is identified with domain labels. Moreover, we employ $\\textcolor{green}{\\ln{p_{x|{z}}}({x}|{z})=\\ln{p_{x|{z}}}(x|{z}_1,{z}_2,{z}_3,{z}_4)}$ as a constraint on $z_4$, indicating that $z_4$ is with block-wise identification. \n\n$ELBO = E_{q_{z|x}(z|x)}[\\textcolor{green}{ln p_{x|z}(x|z)}+ln p_{y|u,z_{2},z_{3}}(y|u,z_{2},z_{3})+\\textcolor{blue}{\\ln p_{u|z}(u|z)}]-D_{KL}(q_{z|x}(z|x)||p_{z}(z))$\n\nIn the implementation, since the reconstruction of $u$ is not the optimization goal and might bring insignificant complexity of optimization, we remove the reconstruction of $u$ as mentioned in Line 200 of Page 5.\n\nIn light of your suggestions, we further conduct an experiment to evaluate the effectiveness of the reconstruction of $u$. Specifically, we add a domain classifier to our SIG, named SIG+D. Experimental results are shown in the following table. According to the experiment results, we can find that the SIG+D also achieves a comparative performance and the standard SIG can achieve an ideal performance without introducing extra complexity. \n\n||Art|Clipart|Product|RealWorld|Average|\n|-|-|-|-|-|-|\n|SIG|76.4|63.9|85.4|85.8|77.8|\n|SIG+D|76.1|63.2|85.6|85.8|77.7|\n\n>**Q3**: It’s unclear how the datasets, which contain mostly covariate shifts, were used for studying the two other shifts.\n\n**A3**: Thanks a lot for this question. We have tried our best to include the most mainstream benchmark datasets for multi-source domain adaptation.  While most of them follow the covariate shift, some of them consist of other types of domain shift like conditional shift and target shift. For example, in office-home datasets, pictures with monitor, computer, and laptop are tagged with the same label, meaning that the covariate shift assumption ($p_{\\mathcal{S}}(y|x)=p_{\\mathcal{T}}(y|x)$) does not always hold, but conditional shift ($p_{\\mathcal{S}}(x|y) \\neq p_{\\mathcal{T}}(x|y)$) holds. Moreover, $p(y)$ also slightly changes across domains. Therefore, these datasets contain a mixture of three types of shifts instead of covariate shifts. \n\nIn light of your suggestion, we design an experimental setting where the target shift changes across domains. Experiment results on SIG, iMSDA, and WADN are shown in the table below. We find that our method achieves the best performance, evaluating the effectiveness of the target shift scenario.\n\n||Art|Clipart|Product|RealWorld|Average|\n|-|-|-|-|-|-|\n|SIG|75.4|62.0|84.9|85.2|77.8|\n|iMSDA|73.2|57.2|82.6|84.2|74.3|\n|WADN|74.8|60.6|84.1|84.2|75.9|\n\n>**Q4**:  ...For example, in Section 4.2, $\\hat{z}_{3,\\mathcal{S}}^{(i)}$ is denoted as \"..latent variables of $i$-th class from source\". How is this estimated?\n\n**A4**: We sincerely appreciate your suggestion of our paper for better clarity. We estimate $\\hat{z}_{3,\\mathcal{S}}^{(i)}$ via a moving average method as follows:\n\n1) For each class, we initialize $\\hat{z}_{3,\\mathcal{S}}^{(i,0)}$ in the first training step. \n2) In the $\\tau$ training step, we randomly sample a batch of sample $(x_k, y_k)_{k=1}^{B}$ with the size of $B$ from source domain $\\mathcal{S}$. \n3) We estimate $z_3$ of each sample via the pre-trained backbone networks and the bottleneck networks, so we obtain $(z_{3,k}, y_k)_{k=1}^{B}$.\n4) Given the $i$-th class, we calculate $c_{3,\\mathcal{S}}^{(i)}=\\frac{1}{B_i} \\sum \\limits_{(z_{3,k},y_k=i)} z_{3,k}$, where $B_i$ is the sample number of the $i$-th class.\n5) Sequentially, we update the estimated variables via $\\hat{z}_{3,\\mathcal{S}}^{(i,\\tau)}=\\gamma c_{3,\\mathcal{S}}^{(i)} + (1-\\gamma) \\hat{z}_{3,\\mathcal{S}}^{(i,\\tau-1)}$ where $\\gamma$ is a hyper-parameter.  *(We also provided a PDF version of Q4 due to some issues related to the Markdown formatting and LaTeX equations in OpenReview)*\n\nIn light of your valuable advice, we have added an algorithm in Appendix to introduce how to estimate $\\hat{z}_{3,\\mathcal{S}}^{(i)}$ and will include it in the final version.\n\n>**Q5**: The derivation of the ELBO in Equation 5 should be provided to enhance the completeness of the paper.\n\n**A5**: Thank you very much for this suggestion to improve the completeness of our paper. In light of your suggestion, we have provided the derivation of the ELBO in the Appendix, which is shown as follows.\n\n${\\quad}\\ln p(x, y, u) = \\ln \\frac{p(x, y, u, z)}{p(z|x, y, u)}=\\ln \\frac{p(x|z)p(u,y,z)}{p(z|x, y, u)}=\\ln \\frac{p(x|z)p(y|u,z_2,z_3)p(u|z)p(z)}{p(z|x, y, u)}$\n\n$=E_{q(z|x)}\\ln\\frac{p(x|z)p(y|u,z_2,z_3)p(u|z)p(z)}{q(z|x)} + D_{KL}(q(z|x)||p(z|x,y,u))$\n\n$\\geq E_{q(z|x)}\\ln p(x|z) + E_{q(z|x)} \\ln p(y|u,z_2,z_3) + E_{q(z|x)} \\ln p(u|z) - D_{KL}(q(z|x)||p(z)) = ELBO$"}, {"review_id": "zlPzbgduXf", "reviewer": "Reviewer_nEWG", "summary": "In this paper, the authors investigate the problem of multi-source domain adaptation. In detail, the authors first devise a causal generation process, which consider the domain-specific and label-irrelevant, domain-specific and label-relevant, domain-invariant and label-invariant, domain-invariant and label-irrelevant variables. Based on the aforementioned generation process, the authors raise the identification guarantees for the latent variables. The authors also evaluate the proposed methods on several benchmarks and achieve good performance.", "questions": "NO", "limitations": "NON", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors provide an interesting perspective for multi-source domain adaptation, which uses nonlinear ICA to identify the latent variables. Compared with the existing methods, the proposed method relaxes the assumptions.\n2.\tThe authors evaluate the proposed methods on several datasets and achieve the ideal results.\n", "weaknesses": "1.\tAccording to the theoretical results, the authors give three assumptions which is similar to those of nonlinear ICA literature. Why are these existing assumptions applicable in real-world scenarios? Moreover, according to the subspace identification results, the dimension of latent variables is small and restricted by the number of domains, which is hard to be met in practice.    \n2.\tFor the real-world scenario, the number of ground-truth latent variables is unknown, is it reasonable to assume that their number is according to the observed domains?    \n3.\tThe authors equip a cross-attention module to the ResNet101 for the DomainNet dataset, it is suggested the authors provide the experiment results of other datasets, e.g., Office-home, with the proposed cross-attention module.    \n\nMinors:     \nThere are some typos in this paper. For example, the illustrations of Lemma2 in the main and appendix are inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our deep gratitude for the valuable feedback and helpful suggestions provided on our paper, as well as for the time you devoted to reviewing it. Below, we have addressed each of your comments and suggestions.\n\n>**Q1**: According to the theoretical results, the authors give three assumptions that are similar to those of nonlinear ICA literature. Why are these existing assumptions applicable in real-world scenarios? Moreover, according to the subspace identification results, the dimension of latent variables is small and restricted by the number of domains, which is hard to be met in practice.\n\n**A1**: Thanks for your great question. The assumptions used in this paper are reasonable and mild. We explain them one by one. \n\n1) Assumption 1 implies that $p(z|u)$ is smooth and positive, i.e., the domains are changing continuously, for example, the light directions of images are changing continuously.\n\n2) Assumption 2 means that $p(z|u)=\\prod p(z_i|u)$, i.e., changing factors are independent of each other, for example, the light directions of images and the resolution of images are independent.\n\n3) Assumption 3 indicates that the latent variables can be identified when the number of domains is sufficient. Since it is easy to access the data with different distributions. Hence, these assumptions are applicable in real-world scenarios.\n\nIn practice, the changes between different domains can be described via a small dimension, for example, the change of light directions of images can be described via a one-dimension variable. It is a common minimal change assumption in causality[1]. Therefore, the small dimension of latent variables can be met in practice. We will add a discussion about the reasonableness of the assumptions in the final version.\n\n[1] Domain Adaptation with Invariant Representation Learning: What Transformations to Learn? Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, Kun Zhang, NeurIPS2021\n\n\n>**Q2**: For the real-world scenario, the number of ground-truth latent variables is unknown, is it reasonable to assume that their number is according to the observed domains?\n\n**A2**: Thank you very much for the profound question to help us clarify the implementation of our work. Due to the minimal change assumption mentioned above, the dimension of the changing variables is usually small. Besides, since the latent variables $z_2$ are caused by the domains $u$ and $y$, the dimension of $z_2$ is restricted by $|u|\\times|y|-1$, which is large enough and hence is easy to satisfy in real-world scenarios. For example, the DomainNet dataset contains 6 domains and 345 categories of objects, indicating that the dimension of latent variables can be 2070. Therefore, it is reasonable to assume that their number is according to the observed domains.\n\n\n>**Q3**: The authors equip a cross-attention module to the ResNet101 for the DomainNet dataset, it is suggested the authors provide the experiment results of other datasets, e.g., Office-home, with the proposed cross-attention module.\n\n**A3**: We are grateful for your careful review and constructive suggestion to improve the completeness of our experiments. In light of your suggestion, we have provided the experiment results on the Office-home dataset with the cross-attention module, named SIG+CA, which is shown in the table below. We find that the SIG+CA also achieves comparative results, showing the effectiveness of the cross-attention module.\n\n|       |   Art | Clipart | Product | RealWorld | Average |\n|-------|-------|---------|---------|-----------|---------|\n| Source Only   |  64.5 |    52.3 |  77.6 |  80.7 |  68.8 |\n| DANN   |  64.2 |  58.0 |  76.4 |   78.8 |  69.3 |\n| DAN   |  68.2 |  57.9 |  78.4 |   81.9 |   71.6 |\n| DCTN  |  66.9 |  61.8 |  79.2 |   77.7 |  71.4 |\n| MFSAN   | 72.1 |  62.0 |  80.3 |  81.8 |  74.1 |\n| WADN  |  75.2 |    61.0 |  83.5 |      84.4 |    76.1 |\n| iMSDA   |  75.4 |    61.4 |    83.5 |      84.4 |    76.2 |\n| SIG   |  76.4 |    63.9 |    85.4 |      85.8 |    77.8 |\n| SIG+CA |  75.8 |    61.8 |    84.0 |      84.7 |    76.6 |\n\n>**Q4**: There are some typos in this paper. For example, the illustrations of Lemma2 in the main and appendix are inconsistent.\n\n**A4**: Thanks for noticing it and kindly letting us know! We have read the paper carefully and corrected the typos. \n\n\n\n"}, {"review_id": "PHawuxsHvc", "reviewer": "Reviewer_pec6", "summary": "This paper studies the problem of multi source domain adaptation where we have access to multiple labeled source domains and unlabeled target domain. Authors consider a novel data generation process by modeling them through 4 new latent variables (i.e. combinations of domain specific or domain invariant and label relevant/ label irrelevant). This is done to relax the assumptions made by previous theoretical analyses. The authors provide sufficient theoretical analysis in this context by providing subspace identification guarantees and using that to improve the results. They conduct multiple experiments include synthetic experiments and the results are satisfactory. \n", "questions": "Please comment on the weakness I raised above and if you could provide some results even on one dataset where there is either a lesser number of classes in target than in source (https://arxiv.org/abs/1808.04205)  or like in other open set DA problems that would be great\n", "limitations": "Authors have adequately addressed the limitations. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*  The paper is well-presented. \n* The paper provides new framework through their modeling of data generation process by introducing the 4 latent variables and then considering them to influences from the domain index and the target labels. \n* Their theoretical analysis is convincing and experiments are impressive.\n", "weaknesses": "* This framework is somewhat similar to an earlier work on domain generalization(https://arxiv.org/pdf/2102.11436.pdf) . I agree that your data generation process further expands on their structural causal model but noting the similarities it should be cited and expand on the differences\n* Importantly, since we model label relevant attributes separately, should we still make the assumption that source and target contain the same labels. \n* My question specifically is can you comment on if this framework naturally support open-set/partial MSDAs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your valuable comments, helpful suggestions, and encouragement. Below please see our point-to-point responses to your comments and suggestions.\n\n>**Q1**: This framework is somewhat similar to earlier work on domain generalization(https://arxiv.org/pdf/2102.11436.pdf). I agree that your data generation process further expands on their structural causal model but noting the similarities it should be cited and expand on the differences\n\n**A1**: We sincerely appreciate your recommendation on these methods, which help clarify the difference between our contributions and other works. In light of your suggestions, we have included the discussions and comparisons in the revised related work. Several methods [1][2][3] employ the prior causal structure to address the distribution shift challenges. Although these methods leverage causal graphs to investigate how the distributions change, the proposed method employs a different causal generation process, which considers different types of domain shifts and is more general. Moreover, our method considers different types of latent variables and provides subspace identification guarantees.\n\n[1] Model-Based Domain Generalization Alexander Robey, George J. Pappas, Hamed Hassani, NeurIPS2021 \n\n[2] Learning Disentangled Semantic Representation for Domain Adaptation, Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, Zhifeng Hao, IJCAI2019\n\n[3] Domain Adaptation under Target and Conditional Shift, Kun Zhang, Bernhard Sch¨olkopf, Krikamol Muandet, Zhikun Wang, ICML2013 \n\n>**Q2, Q3, Q4:** Importantly, since we model labels relevant attributes separately, should we still make the assumption that source and target contain the same labels?\n My question specifically is can you comment on if this framework naturally supports open-set/partial MSDAs.\n Please comment on the weakness I raised above and if you could provide some results even on one dataset where there is either a lesser number of classes in target than in source (https://arxiv.org/abs/1808.04205) or like in other open set DA problems that would be great\n\nA: Thanks for your treasure suggestions. We would like to highlight that the proposed SIG is a general framework for multi-source domain adaptation. In this paper, we employ the standard multi-source domain adaptation setting to evaluate our method, where the label space of the source and target domain is assumed to be the same. Generally speaking, our method can extend to other scenarios, such as open-set MSDAs and partial MSDAs. As you mentioned, it is because we model labels relevant attributes separately, and hence allow the setting where source and target domains contain different labels.\n\nFor example, in partial MSDAs[4], the source label space is a superset of the target label space. One of the challenges of partial MSDAs is to mitigate the influence of the source labeled data in outlier label space. In light of your suggestion, we have extended our method to the partial domain MSDAs setting, following the paradigm of PADA[4] and using the pseudo label to estimate the target label space. Specifically, we apply the class-aware conditional alignment on the target label space, where the confidence of the source-specific label is low. Moreover, we have provided the experimental results of the partial MSDAs in the following table, where five labels are removed in the target domain to satisfy the partial MSDA setting. We find that our SIG model achieves superior performance. \n\n|       |   Art | Clipart | Product | RealWorld | Average |\n|-------|-------|---------|---------|-----------|---------|\n| SIG |  76.0 |    62.7 |    85.8 |      85.6 |    77.5 |\n| PSDA  |  75.5 |    60.4 |    83.3 |      83.8 |    75.7 |\n\n[4] Partial Adversarial Domain Adaptation, Zhangjie Cao, Lijia Ma, Mingsheng Long, Jianmin Wang,ECCV2018\n"}, {"review_id": "hiOaXPzK8P", "reviewer": "Reviewer_bTZb", "summary": "This paper considered multiple-source domain adaptation problems in the context of causal representation learning. Specifically, this paper considered quite general scenarios in the domain adaptation by figuring four different representations.\n\n- Domain specific and label irrelevant \n- Domain specific and label relevant\n- Domain invariant and label irrelevant \n- Domain invariant and label relevant\n\nNotably, this paper derived theoretical assumptions via illustrating when we could identify these four representations (or subspace). A VAE based approach in deep learning is further proposed to identify the subspace. Empirical results are further validated in synthetic data, benchmark domain adaptation dataset (including challenging DomainNET dataset)\n", "questions": "See comments in the weakness part.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this reviewer feels that this paper has a solid contribution in the theoretical aspects in causal based domain adaptation. Extensive experiments further validated the practical utility. \n\n- **Significance & Originality**\nThis paper proposes a very solid theoretical justification in the identifiability issue in multi-source da, which is a fundamental problem. More importantly, this paper studied the general setting (or most difficult settings) in multi-source da. Besides, theoretical assumptions are proposed, which is generally non-trivial.\n\n- **Quality**\nThe theoretical assumptions and proof seem valid and reasonable for me. The proposed practical method is also reasonable. Empirical results in toy data and real-data demonstrated that the proposed method could better identify different representations. \n\n- **Clarity**\nIn general, this paper is clearly written and well-explained. Since this paper is a bit theoretical, there are still several theoretical points that need to be clarified (see the weakness part). \n", "weaknesses": "Since this paper covers several fundamental proofs, this reviewer feels uncertain about several points within the paper. \n\n\n1. Theorem 1 about theoretical assumptions. In general, I could follow the proof, where intuitions within these assumptions could be better explained. I just take theorem 1 as an example.\n\n- About conditional independence. If I understand correctly, this implies, the representation variable $P(Z|U) = \\prod P(z_i|U)$ ?  (sort of disentanglement or mean-field approximation assumption)? If we write it as a log probability term, it will be $\\log P(Z|U) = \\sum \\log P(Z_i|U)$, which is consistent with your paper…\n- About linear independence. This assumption seems to illustrate that the gradient of the probability density over each component should be linearly independent? That implies if we use a gradient based approach to learn each component, each component will converge to a distinct direction. Thus we could possibly identify different independent components?\n- I have a general question in theory 1 (for example). Is it possible to extend this assumption into the blockwise vectors? i.e, $P(Z|U) = P(Z_1|U)P(Z_2|U)P(Z_3|U)P(Z_4|U)$, where $Z_1$ is a subspace vector rather than a scalar. \n\n2. About explanations in toy data. It seems that the proposed method works quite significantly in toy data. However the experimental details (such as the data generation) could be better clarified. For example, is there some sort of spurious correlation?\n\n3. In Equation (1) Line 3, I would think this holds when we have figure 1 by considering the conditional independence. \n\n4. I like the example in explaining different subspace in line 82-92. It could be better to consider this example into more real-world scenarios such as drug discovery and health. For example, data collected from different hospitals may create different $Z_1$, this could be helpful to better understand the importance of considering a general scenario. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your valuable comments, helpful suggestions, and encouragement. Below please see our point-to-point responses to your comments and suggestions.\n\n>**Q1**: Theorem 1 about theoretical assumptions. In general, I could follow the proof, where intuitions within these assumptions could be better explained. I just take theorem 1 as an example. \n\n>1) About conditional independence. If I understand correctly, this implies, the representation variable $P(z|u)=\\prod(z_i|u)$\n ? (sort of disentanglement or mean-field approximation assumption)? If we write it as a log probability term, it will be $\\log P(z|u)=\\sum P(z_i|u)$, which is consistent with your paper…\n>2) About linear independence. This assumption seems to illustrate that the gradient of the probability density over each component should be linearly independent. That implies if we use a gradient-based approach to learn each component, each component will converge in a distinct direction. Thus we could possibly identify different independent components.\n>3) I have a general question in theory 1 (for example). Is it possible to extend this assumption into the blockwise vectors? i.e, $P(z|u)=P(z_1|u)P(z_2|u)P(z_3|u)P(z_4|u)$, where $z_1$ is a subspace vector rather than a scalar.\n\n**A1**: I would like to express my sincere gratitude for your meticulous review. We answer these questions point by point.\n\n1) s for conditional independence, I think that the understanding of the reviewer, where the representation variables follow the mean-field approximation assumption, is correct. The intuition of conditional independence is that the changing factors are independent of each other. For example, in the image classification scenario, the light directions of images and the resolution of images change independently.\n2) I think there is some misunderstanding between the derivative of latent variables and the derivative of model parameters. On one hand, the linear independence assumption implies that the $w(z,u_j)-w(z,u_0)$ are linearly independent, and each element in $w(z,u_j)=(\\frac{\\partial q_1(z_1,u)}{\\partial z_1},\\cdots, \\frac{\\partial q_i(z_i,u)}{\\partial z_i},\\cdots \\frac{\\partial q_{n_s}(z_{n_s},u)}{\\partial z_{n_s}})$ denotes the first-order derivative of the ground truth log probability density $q_i(z_i,u)$ with respect to ground-truth latent variables $z_i$. On the other hand, the gradient of the gradient-based approaches is calculated by the derivative of the loss with respect to the parameters of the models. These two derivatives are different.\n3) Existing disentanglement methods based on nonlinear ICA usually assume that each dimension of latent variables is independent of each other. However, there are some cases that the independent assumption can not be met. So disentanglement under dependent latent variables ($p(z|u)=P(z_1|u)P(z_2|u)P(z_3|u)P(z_4|u)$ is a special case of dependent latent variables) is a future direction. In this scenario, homogeneous linear equations are hard to be developed since the latent variables are not independent, but the property of independence of latent variables given their Markov Blanket can be used to develop the homogeneous linear equations.\n\n>**Q2**: About explanations in toy data. It seems that the proposed method works quite significantly in toy data. However, the experimental details (such as the data generation) could be better clarified. For example, is there some sort of spurious correlation?\n\n**A2**: In light of your valuable suggestions, we have reorganized the causal generation process. In detail, we generate the simulation data for binary classification with 8 domains, which follows the data generation process as shown in Figure 2, which includes two types of latent variables, i.e., domain-specific latent variables $z_s$ and domain-invariant latent variables $z_c$. To better evaluate the subspace identification results, we let the label distribution is the same across different domains, so there is no spurious correlation. The domain-specific latent variables $z_s$ are sampled from 8 different mixtures of Gaussians and $z_c$ are sampled from a factorized Gaussian distribution. As for the nonlinear generation process, we let the observed variables be generated via an MLP with the Tanh activation function.\n\n\n>**Q3**: In Equation (1) Line 3, I would think this holds when we have Figure 1 by considering conditional independence.\n\n**A3**: Thank you for your reminder. We totally agree with you, and leverage the property of the causal generation process when deriving Equation (1). In light of your suggestion, we have provided a clear explanation of how to derive Equation (1). In detail, the derivation in Equation (1) can be separated into three steps. \n\n1) We introduce the latent variables $z_1,z_2,z_3$, and $z_4$, which have mentioned in Section 2.1. \n2) We factorize the joint distribution in Equation (1) into $p_{x,z_1,z_2,z_3,z_4|y,u_{\\mathcal{T}}}$ and $p_{y|u_{\\mathcal{T}}}$ with the help of Bayes Rule. \n3) We further use Bayes Rule to factorize $p_{x,z_1,z_2,z_3,z_4|y,u_{\\mathcal{T}}}$. Since $x$ is conditional independent of $u, y$ given $z_1,z_2,z_3,z_4$, we can obtain $p_{x|z_1,z_2,z_3,z_4}$.\n\n>**Q4**: ...It could be better to consider this example in more real-world scenarios such as drug discovery and health. For example, ....\n\n**A4**: Thank you for your affirmation. In the scenario of health, we try to provide an example of malaria detection, $z_1$ denotes the domain-specific information, for example, the children's hospital and the contagious hospital; $z_2$ denotes the latent variables that can be age relevant to domains and labels. For example, infants are more likely to suffer from malaria, and the age of patients are also influenced by the different hospital; $z_3$ denotes the symptoms of patients; And $z_4$ denotes the label-irrelevant latent variables like gender.\n"}]

Author rebuttal:
Dear Reviewers Yx4K, nEWG, pec6, and bTZb:

Thanks for the thoughtful and constructive review. It is encouraging that the reviewers think SIG is novel (Reviewer Yx4K and pec6), interesting (Reviewer Yx4K and nEWG), and solid ( Reviewer bTZb). We here provide a general response to summarize the modifications of the paper.

- To Reviewer Yx4K, we have clarified the intuition of the "domain-invariant and label-irrelevant" latent variables $z_4$.
- To Reviewer Yx4K, we have explained the identifiability of $z_1$ and $z_4$, and provided experimental results according to your suggestions.
- To Reviewer Yx4K, we have clarified how the datasets are used to study the conditional shift and target shift. And we also added experimental results in light of your suggestions.
- To Reviewer Yx4K, we have added the derivation of the ELBO in Equation (5).
- To Reviewer nEWG, we have explained the reasonableness of the assumptions.
- To Reviewer nEWG, we have explained the assumptions and the reasonableness for the number of latent variables.
- To Reviewer nEWG, we have added the experimental results of our SIG with the cross-attention module of the OfficeHome dataset.
- To Reviewer nEWG, we have read the paper carefully and corrected the typos.
- To Reviewer pec6, we have clarified the difference between our contributions and other works as your recommended.
- To Reviewer pec6, we have discussed the assumption of the same label space of source and target domains.
- To Reviewer pec6, we have commented on the extension to open-set/partial MSDA of our method and added experimental results of partial MSDA.
 - To Reviewer bTZb, we have clarified the intuition of assumptions.
 - To Reviewer bTZb, we have discussed the conditional independence of Equation (1).
 - To Reviewer bTZb, we have provided an example in the health scenario.

Thanks again for your time dedicated to carefully reviewing this paper. We hope that our response properly addresses your concerns.

With best regards, 

Authors of submission 4882


Meta-review: This paper received strong reviews with one strong accept and three accept recommendations. The paper addresses the problem of multi-source unsupervised domain adaptation and focuses on reducing the assumptions required in prior methods (invariant label distributions, large number of domains etc). The proposed method of subspace identification guarantee (SIG) was found to be novel, interesting, and theoretically grounded by the reviewers. Further, the method achieved state of the art performance across four standard datasets. The rebuttal/discussion between the authors and reviewers was productive with many of the reviewers questions being answered and positively impacting their final recommendations. Given the strength of this work and unique direction it proposes, the AC recommends this work be highlighted at the conference.
Decision: Accept (spotlight)