Title: Subspace Identification for Multi-Source Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain Adaptation

Identification
Abstract: Multi-source domain adaptation (MSDA) methods aim to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Although current methods achieve target joint distribution identifiability by enforcing minimal changes across domains, they often necessitate stringent conditions, such as an adequate number of domains, monotonic transformation of latent variables, and invariant label distributions. These requirements are challenging to satisfy in real-world applications. To mitigate the need for these strict assumptions, we propose a subspace identification theory that guarantees the disentanglement of domain-invariant and domain-specific variables under less restrictive constraints regarding domain numbers and transformation properties and thereby facilitating domain adaptation by minimizing the impact of domain shifts on invariant variables. Based on this theory, we develop a Subspace Identification Guarantee (SIG) model that leverages variational inference. Furthermore, the SIG model incorporates class-aware conditional alignment to accommodate target shifts where label distributions change with the domain. Experimental results demonstrate that our SIG model outperforms existing MSDA techniques on various benchmark datasets, highlighting its effectiveness in real-world applications.

Subspace Identification for Multi-Source Domain Adaptation

Abstract

Multi-source domain adaptation (MSDA) methods aim to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Although current methods achieve target joint distribution identifiability by enforcing minimal changes across domains, they often necessitate stringent conditions, such as an adequate number of domains, monotonic transformation of latent variables, and invariant label distributions. These requirements are challenging to satisfy in real-world applications. To mitigate the need for these strict assumptions, we propose a subspace identification theory that guarantees the disentanglement of domain-invariant and domain-specific variables under less restrictive constraints regarding domain numbers and transformation properties, thereby facilitating domain adaptation by minimizing the impact of domain shifts on invariant variables. Based on this theory, we develop a Subspace Identification Guarantee (SIG) model that leverages variational inference. Furthermore, the SIG model incorporates class-aware conditional alignment to accommodate target shifts where label distributions change with the domains. Experimental results demonstrate that our SIG model outperforms existing MSDA techniques on various benchmark datasets, highlighting its effectiveness in real-world applications.

1 Introduction

Multi-Source Domain Adaptation (MSDA) is a method of transferring knowledge from multiple labeled source domains to an unlabeled target domain, to address the challenge of domain shift between the training data and the test environment. Mathematically, in the context of MSDA, we assume the existence of M source domains {S 1 , S 2 , ..., S M } and a single target domain T . For each source domain S i , data are drawn from a distinct distribution, represented as p x , y | u S i , where the variables x , y , u correspond to features, labels, and domain indices, respectively. In a similar manner, the distribution within the target domain T is given by p x , y | u T . In the source domains, we have access to m i annotated feature-label pairs of each domain, denoted by ( x S i , y S i ) = ( x S i k , y S i k ) m i k =1 , while in the target domain, only m T unannotated features are observed, represented as ( x ( T ) ) = ( x T k ) m T k =1 . The primary goal of MSDA is to effectively leverage these labeled source data and unlabeled target data to identify the target joint distribution p x , y | u T .

However, identifying the target joint distribution of x , y | u T using only x | u T as observations present a significant challenge, since the possible mappings from p x , y | u T to p x | u T are infinite when no extra constraints are given. To solve this problem, some assumptions have been proposed to constrain the domain shift, such as covariate shift [44], target shift [58, 1], and conditional shift [3, 56]. For example, the most conventional covariate shift assumption posits that p y | x is fixed across different domains

while p x varies. Under this assumption, researchers can employ techniques such as importance reweighting [52], invariant representation learning [11], or cycle consistency [17] for distribution alignment. Additionally, target shift assumes that p x | y is fixed while the label distribution p y changes, whereas conditional shift is represented by a fixed p y and a varying p x | y . More generally, the minimal change principle has been proposed, which not only unifies the aforementioned assumptions but also enables the theoretical guarantee of the identifiability of the target joint distribution. Specifically, it assumes that p y | u and p x | y , u change independently and the change of p x | y , u is minimal. Please refer to Appendix G for further discussion of related works, including domain adaptation and identification.

Although current methods demonstrate the identifiability of the target joint distribution through the minimal change principle, they often impose strict conditions on the data generation process and the number of domains, limiting their practical applicability. For instance, iMSDA [30] presents the component-wise identification of the domain-changed latent variables, subsequently identifying target joint distribution by modeling a data generation process with variational inference. However, this identification requires the following conditions. First, a sufficient number of auxiliary variables is employed for the component-wise theoretical guarantees, meaning that when the dimension of latent variables is n , a total of 2 n +1 domains are needed. Second, in order to identify high-level invariant latent variables, a component-wise monotonic function between latent variables must be assumed. Third, these methods implicitly assume that label distribution remains stable across domains, despite the prevalence of target shift in real-world scenarios. These conditions are often too restrictive to be met in practice, highlighting the need for a more general approach to identifying latent variables across a wider range of domain shifts.

In an effort to alleviate the need for such strict assumptions, we present a subspace identification theory in this paper that guarantees the disentanglement of domain-invariant and domain-specific variables under more relaxed constraints concerning the number of domains and transformation properties. In contrast to component-wise identification, our subspace identification method demands fewer auxiliary variables (i.e., when the dimension of latent variables is n , only n +1 domains are required). Additionally, we design a more general data generation process that accounts for target shift and does not necessitate monotonic transformation between latent variables. In this process, we categorize latent variables into four groups based on whether they are influenced by domain index or label. Building on the theory and causal generation process, we develop a Subspace Identification Guarantee (SIG) model that employs variational inference to identify latent variables. A class-aware condition alignment is incorporated to mitigate the impact of target shift, ensuring the update of the most confident cluster embedding. Our approach is validated through a simulation experiment for subspace identification evaluation and four widely-used public domain adaptation benchmarks for application evaluation. The impressive performance that outperforms state-of-the-art methods demonstrates the effectiveness of our method.

2 Identifying Target Joint Distribution with Data Generation Process

2.1 Data Generation Process

We begin with introducing the data generation process. As shown in Figure 1, the observed data x ∈ X are generated by latent variables z ∈ Z ⊆ R n . Sequentially, we divide the latent variables z into the four parts, i.e. z = { z 1 , z 2 , z 3 , z 4 } ∈ Z ⊆ R n , which are shown as follows.

domain-specific and label-irrelevant variables z 1 ∈ R n 1 .

domain-invariant and label-relevant variables z 3 ∈ R n 3 .

domain-specific but label-relevant variables z 2 ∈ R n 2 .

domain-invariant but label-irrelevant variables z 4 ∈ R n 4 .

To better understand these latent variables, we provide some examples in DomainNet datasets. First, z 1 ∈ R n 1 denotes the styles of the images like 'infograph' and 'sketch', which are irrelevant to labels. z 2 ∈ R n 2 denotes the latent variables that can be the texture information relevant to domains and labels.

ଵ

ଶ

ଷ

ସ

For example, the samples of 'clock' and 'telephone' contain some digits, and these digits in these samples are a special texture, which can be used for classification and be influenced by different

styles, such as 'infograph'. z 3 ∈ R n 3 denotes the latent variables that are only relevant to the labels. For example, in the DomainNet dataset, it can be interpreted as the meaning of different classes like 'Bicycle' or 'Teapot'. Finally, z 4 ∈ R n 4 denotes the label-irrelevant latent variables. For example, z 4 can be interpreted as the background that is invariant to domains and labels.

Based on the definitions of these latent variables, we let the observed data be generated from z through an invertible and smooth mixing function g : Z → X . Due to the target shift, we further consider that the p y is influenced by u , i.e. u → y .

Compared with the existing data generation process like [30], the proposed data generation process is different in three folds. First, p u is independent of p y in the iMSDA [30], so the target shift is not taken into account. Second, the data generation process of iMSDA requires an invertible and monotonic function between latent variables for component-wise identification, which is too strict to be met in practice. Third, to provide a more general way to depict the real-world data, our data generation process introduces the domain-specific but label-relevant latent variables z s 2 and the domain-invariant but label-irrelevant variables z 4 .

2.2 Identifying the Target Joint Distribution

In this part, we show how to identify the target joint distribution p x , y | u T with the help of marginal distribution. By introducing the latent variables and combining the proposed data generation process, we can obtain the following derivation.

According to the derivation in Equation (1), we can identify the target joint distribution by modeling three distributions. First, we need to model p x | z 1 , z 2 , z 3 , z 4 , implying that we need to model the conditional distribution of observed data give latent variables, which coincides with a generative model for observed data. Second, we need to estimate the label pseudo distribution of target domain p y | u T . Third, we need to model p z 1 , z 2 , z 3 , z 4 | y , u T meaning that the latent variables should be identified with auxiliary variables u , y under theoretical guarantees. In the next section, we will introduce how to identify these latent variables with subspace identification block-wise identification results.

3 Subspace Identifiability for Latent Variables

In this section, we provide how to identify the latent variables in Figure 1. In detail, we first prove that z 2 is subspace identifiable and z 1 , z 3 can be reconstructed from the estimated ˆ z 1 , ˆ z 2 , ˆ z 3 . Then we further prove that z 4 is block-wise identifiable.

To clearly introduce the subspace identification theory, we employ a simple data generation process [3] as shown in Figure 2. In this data generation process, z s ∈ R n s and z c ∈ R n c denote the domain-specific and domain-invariant latent variables, respectively. For convenient, we let z = { z s , z c } , n = n s + n c . Moreover, we assume z s = ( z i ) n s i =1 and z c = ( z i ) n i = n s +1 . And { u , y , x } denote the domain index, labels, and observed data, respectively. And we further let the observed data be generated from z through an invertible and smooth mixing function g : Z → X . The subspace identification of z s means that for each ground-truth z s,i , there exits ˆ z s and an invertible function h i : R n → R , such that z s,i = h i (ˆ z s ) .

௦

௖

Theorem 1. ( Subspace Identification of z s . ) We follow the data generation process in Figure 2 and make the following assumptions:

A1 (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e., p z | u > 0 over Z and U .

̸

A2 (Conditional independent): Conditioned on u , each z i is independent of any other z j for i, j ∈ { 1 , · · · , n } , i = j , i.e. log p z | u ( z | u ) = ∑ n i q i ( z i , u ) where q i ( z i , u ) is the log density of the conditional distribution, i.e., q i : log p z i | u .

A3 (Linear independence): For any z s ∈ Z s ⊆ R n s , there exist n s +1 values of u , i.e., u j with j = 0 , 1 , · · · , n s , such that these n s vectors w ( z , u j ) -w ( z , u 0 ) with j = 1 , · · · , n s are linearly independent, where vector w ( z , u j ) is defined as follows:

By modeling the aforementioned data generation process, z s is subspace identifiable.

Proof sketch. First, we construct an invertible transformation h between the ground-truth z and estimated ˆ z . Sequentially, we leverage the variance of different domains to construct a full-rank linear system, where the only solution of ∂ z s ∂ ˆ z c is zero. Since the Jacobian of h is invertible, for each z s,i , i ∈ { 1 , · · · , n s } , there exists a h i such that z s,i = h i (ˆ z ) and z s is subspace identifiable.

The proof can be found in Appendix B.1. The first two assumptions are standard in the componentwise identification of existing nonlinear ICA [30, 27]. The third Assumption means that p z | u should vary sufficiently over n +1 domains. Compared to component-wise identification, which necessitates 2 n +1 domains and is likely challenging to fulfill, subspace identification can yield equivalent results in terms of identifying the ground-truth latent variables with only n +1 domains. Therefore, subspace identification benefits from a more relaxed assumption.

Based on the theoretical results of subspace identification, we show that the ground-truth z 1 , z 2 and z 3 be reconstructed from the estimated ˆ z 1 , ˆ z 2 and ˆ z 3 . For ease of exposition, we assume that z 1 , z 2 , z 3 , and z 4 correspond to components in z with indices { 1 , · · · , n 1 } , { n 1 +1 , · · · , n 1 + n 2 } , { n 1 + n 2 + 1 , · · · , n 1 + n 2 + n 3 } , and { n 1 + n 2 + n 3 +1 , · · · , n } , respectively.

Corollary 1.1. We follow the data generation in Section 2.1, and make the following assumptions which are similar to A1-A3:

A4 (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e., p z | u , y > 0 over Z , U , and Y .

̸

A5 (Conditional independent): Conditioned on u and y , each z i is independent of any other z j for i, j ∈ { 1 , · · · , n } , i = j , i.e. log p z | u , y ( z | u , y ) = ∑ n i q i ( z i , u , y ) where q i ( z i , u , y ) is the log density of the conditional distribution, i.e., q i : log p z i | u , y .

A6 (Linear independence): For any z ∈ Z ⊆ R n , there exists n 1 + n 2 + n 3 + 1 combination of ( u , y ) , i.e. j = 1 , · · · , U and c = 1 , · · · , C and U × C -1 = n 1 + n 2 + n 3 , where U and C denote the number of domains and the number of labels. such that these n 1 + n 2 + n 3 vectors w ( z , u j , y c ) -w ( z , u 0 , y 0 ) are linearly independent, where w ( z , u j , y c ) is defined as follows:

By modeling the data generation process in Section 2.1, z 2 is subspace identifiable, and z 1 , z 3 can be reconstructed from ˆ z 1 , ˆ z 2 and ˆ z 2 , ˆ z 3 , respectively.

Proof sketch. The detailed proof can be found in Appendix B.2. First, we construct an invertible transformation h to bridge the relation between the ground-truth z and the estimated ˆ z . Then, we repeatedly use Theorem 1 three times by considering the changing of labels and domains. Hence, we find that the values of some blocks of the Jacobian of h are zero. Finally, the Jacobian of h can be formalized as Equation (4).

where J h denotes the Jacobian of h and J ij h := ∂ z i ∂ ˆ z j and i, j ∈ { 1 , 2 , 3 , 4 } . Since h ( · ) is invertible, J h is a full-rank matrix. Therefore, for each z 2 ,i , i ∈ { n 1 + 1 , · · · , n 1 + n 2 } , there exists a h 2 ,i such that z 2 ,i = h i (ˆ z 2 ) . Moreover, for each z 1 ,i , i ∈ { 1 , · · · , n 1 +1 } , there exists a

Pre-trained

Backbone

Networks

Bottleneck

Networks

𝑓 �

𝑓 �

Decoder

𝑧�

𝑧�

𝑧�

𝑧�

x

𝑺

x

𝑻

𝑓 �

,

𝑓 �

𝑓 �

�

,

𝑓 �

�

Label

Predictor

Class-aware

Condition

Alignment

𝒚 �

𝑺

𝑞�

|

�

𝑝�

|

�

,

��

,

��

𝑞�

|

�

𝑢

h 1 ,i such that z 1 ,i = h 1 ,i (ˆ z 1 , ˆ z 2 ) . And for each z 3 ,i , i ∈ { n 1 + n 2 +1 , · · · , n 1 + n 2 + n 3 } , there exists a h 3 ,i such that z 3 ,i = h 3 ,i (ˆ z 2 , ˆ z 3 ) . Then we prove that z 4 is block-wise identifiable, which means that there exists an invertible function h 4 , s.t. z 4 = h 4 (ˆ z 4 ) .

Lemma 2. [30] Following the data generation process in Section 2.1 and the assumptions A4-A6 in Theorem 3, we further make the following assumption:

A7 (Domain Variability: For any set A z ⊆ Z ) with the following two properties: 1) A z has nonzero probability measure, i.e. P [ z ∈ A z |{ u = u ′ , y = y ′ } ] > 0 for any u ′ ∈ U and y ′ ∈ Y . 2) A z cannot be expressed as B z 4 ×Z 1 ×Z 2 ×Z 3 for any B z 4 ⊂ Z 4 .

̸

∃ u 1 , u 2 ∈ U and y 1 , y 2 ∈ Y , such that ∫ z ∈A z p z | u 1 , y 1 d z = ∫ z ∈A z p z | u 2 , y 2 d z . By modeling the data generation process in Section 2.1, the z 4 is block-wise identifiable.

The proof can be found in Appendix B.3. Lemma 4 shows that z 4 can be block-wise identifiable when the p z | u changes sufficiently across domains.

In summary, we can obtain the estimated latent variables ˆ z with the help of subspace identification and block-wise identification.

4 Subspace Identification Guarantee Model

Based on the theoretical results, we proposed the Subspace Identification Guaranteed model (SIG) as shown in Figure 3, which contains a variational-inference-based neural architecture to model the marginal distribution and a class-aware conditional alignment to mitigate the target shift.

4.1 Variational-Inference-based Neural Architecture

According to the data generation process in Figure 1, we first derive the evidence lower bound (ELBO) in Equation (5).

Since the reconstruction of u is not the optimization goal, we remove the reconstruction of u and we rewrite Equation (5) as the objective function in Equation (6).

To minimize the pairwise class confusion, we further employ the minimum class confusion [25] into the classification loss L y . According to the objective function in Equation (6), we illustrate how to

implement the SIG model in Figure 3. First, we take the observed data x S i and x T from the source domains and the target domain as the inputs of the pre-trained backbone networks like ResNet50 and extract the feature f S i and f T . Sequentially, we employ an MLP-based encoder, which contains bottleneck networks, f µ and f σ , to extract the latent variables z . Then, we take z to reconstruct the pre-trained features via an MLP-based decoder to estimate the marginal distribution p x | u . Finally, we take the z 2 , z 3 , and the domain embedding to predict the source label to estimate p y | u , z 2 , z 3 .

4.2 Class-aware Conditional Alignment

To estimate the target label distribution p y | u T and mitigate the influence of target shift, we propose the class-aware conditional alignment to automatically adjust the conditional alignment loss for each sample. Formally, it can be written as

where C denotes the class number; ˆ z ( i ) 3 , S and ˆ z ( i ) 3 , T denote the latent variables of i th class from source and target domain, respectively; w ( i ) denotes the prediction uncertainty of each class in the target domain; p ˆ y ( i ) denotes the estimated label probability density of i th class; H denotes the entropy.

The aforementioned class-aware conditional alignment is based on the existing conditional alignment loss, which can be formalized in Equation (8).

However, conventional conditional alignment usually suffers from two drawbacks including misestimated centroid and low-quality pseudo-labels. First, the conditional alignment method implicitly assumes that the feature centroids from different domains are the same. But it is hard to estimate the correct centroids of the target domain for the class with low probability density. Second, conditional alignment heavily relies on the quality of the pseudo label. But existing methods usually use pseudo labels without any discrimination, which might result in false alignment. To solve these problems, we consider two types of reweighting.

Distribution-based Reweighting for Misestimated Centroid: Although the conditional alignment method implicitly assumes that the feature centroids from different domains are the same, it is hard to estimate the correct centroids of the target domain for the class with low probability density. To address this challenge, one straightforward solution is to consider the label distribution of the target domain. To achieve this, we employ the technique of black box shift estimation method (BBSE) [37] to estimate the label distribution from the target domain p ˆ y . So we use the estimated label distribution to reweight the conditional alignment loss in Equation (8).

Entropy-based Reweighting for Low-quality Pseudo-labels: conditional alignment heavily relies on the quality of the pseudo label. However, existing methods usually use pseudo labels without any discrimination, which might result in false alignment. To address this challenge, we consider the prediction uncertainty of each class in the target domain. Technologically, for each sample in the target dataset, we calculate the entropy-based weights via the prediction results which are shown as w ( i ) in Equation (7).

By combining the distribution-based weights and the entropy-based weight, we can obtain the class-aware conditional alignment as shown in Equation (7). Hence the total loss of the Subspace Identification Guarantee model can be formalized as follows:

where α, β denote the hyper-parameters.

5 Experiments

5.1 Experiments on Simulation Data

In this subsection, we illustrate the experiment results of simulation data to evaluate the theoretical results of subspace identification in practice.

5.1.1 Experimental Setup

Data Generation. We generate the simulation data for binary classification with 8 domains. To better evaluate our theoretical results, we follow the data generation process in Figure 2, which includes two types of latent variables, i.e., domain-specific latent variables z s and domain-invariant latent variables z c . We let the dimensions of z s and z c be both 2. Moreover, z s are sampled from u different mixture of Gaussians, and z c are sampled from a factorized Gaussian distribution. We let the data generation process from latent variables to observed variables be MLPs with the Tanh activation function. We further split the simulation dataset into the training set, validation set, and test set.

Evaluation Metrics. First, we employ the accuracy of the target domain data to measure the classification performance of the model. Second, we compute the Mean Correlation Coefficient (MCC) between the ground-truth z s and the estimated ˆ z s on the test dataset to evaluate the componentwise identifiability of domain-specific latent variables. A higher MCC denotes the better identification performance the model can achieve. Third, to evaluate the performance of subspace identifiability of domain-specific latent variables, we first use the estimated ˆ z s from the validation set to regress each dimension of the ground-truth z s from the validation set with the help of a MLPs. Sequentially, we take the ˆ z s from the test set as input to estimate how well the MLPs can reconstruct the ground-truth z s from the test set, so we employ Root Mean Square Error (RMSE) to measure the extent of subspace identification. A low RMSE denotes that there exists a transformation h i between z s,i and ˆ z s, 1 , ˆ z s, 2 , i.e. z s,i = h i (ˆ z s, 1 , ˆ z s, 2 ) , i ∈ { 1 , 2 } . For the scenario where the number of domains is less than 8, we first fix the target domain and then try all the combinations of the source domains. And we publish the average performance of all the combinations. We repeat each experiment over 3 random seeds.

5.1.2 Results and Discussion

The experimental results of the simulation dataset are shown in Table 1. According to the experiment results, we can obtain the following conclusions:

1) We can find that the values of MCC increase along with the number of domains. Moreover, the values of MCC are high (around 0.9) and stable when the number of domains is larger than 5. This result corresponds to the theoretical result of component-wise identification, where a certain number of domains (i.e. 2 n +1 ) are necessary for component-wise identification. 2) We can find that the values of RMSE decrease along with the number of domains. Furthermore, the values of RMSE are low and stable (less than 0.07) when the number of domains is larger than 3, but it drops sharply when u = 2 . These experimental results coincide with the theoretical results of subspace identification as well as the intuition where a certain number of domains are necessary for subspace identification (i.e. n s + 1 ). 3) According to the experimental results of ACC, we can find that the accuracy grows along with the number of domains and its changing pattern is relevant to that of RMSE, i.e., the performance is stable when the number of domains is larger than 3. The ACC results also indirectly support the results of subspace identification, since one straightforward understanding of subspace identification is that the domain-specific information is preserved in ˆ z s . And the latent variables are well disentangled with the help of subspace identification, which benefits the model performance.

5.2 Experiments on Real-world Data

5.2.1 Experimental Setup

Datasets: We consider four benchmarks: Office-Home, PACS, ImageCLEF, and DomainNet. For each dataset, we let each domain be a target domain and the other domains be the source domains. For the DomainNet dataset, we equip a cross-attention module to the ResNet101 backbone networks for

better usage of domain knowledge. We also employ the alignment of MDD [78]. For the Office-Home and ImageCLEF datasets, we employ the pre-trained ResNet50 with an MLP-based classifier. For the PACS dataset, we use ResNet18 with an MLP-based classifier. The implementation details are provided in the Appendix C. We report the average results over 3 random seeds.

Baselines: Besides the classical approaches for single source domain adaptation like DANN [11], DAN [40], MCD [50], and ADDA [61]. We also compare our method with several state-of-the-art multi-source domain adaptation methods, for example, MIANγ [45], T-SVDNet [33], LtC-MSDA [63], SPS [64], and PFDA [10]. Moreover, we further consider the WADN [54], which is devised for the target shift of multi-source domain adaptation. For a fair comparison, we employ the same pre-train backbone networks instead of the pre-trained features for WADN in the original paper. We also consider the latest

iMSDA [30], which addresses the MSDA via component-wise identification.

5.2.2 Results and Discussion

Experimental results on Office-Home, ImageCLEF, PACS, and DomainNet are shown in Table 2, 3, and 4, respectively. Experiment results of other compared methods are provided in Appendix D.2.

According to the experiment results of the Office-Home dataset on the left side of Table 2, our SIG model significantly outperforms all other baselines on all the transfer tasks. Specifically, our method outperforms the most competitive baseline by a clear margin of 1 . 3% -4% and promotes the classification accuracy substantially on the hard transfer task, e.g. Clipart. It is noted that our method achieves a better performance than that of WADN, which is designed for the target shift scenario. This is because our method not only considers how the domain variables influence the distribution of labels but also identifies the latent variables of the data generation process. Moreover, our SIG method also outperforms iMSDA, indirectly reflecting that the proposed data generation process is closer to the real-world setting and the subspace identification can achieve better disentanglement performance under limited auxiliary variables.

For datasets like ImageCLEF and PACS, our method also achieves the best-averaged results. In detail, we achieved a comparable performance in all the transfer tasks in the ImageCLFE dataset. In the PACS dataset, our SIG method still performs better than the latest compared methods like iMSDA and T-SVDNet in some challenging transfer tasks like Cartoon. Finally, we also consider the

��

��

��

��

��������

���

��

��

��

�������

��

��

��

��

�������

��

��

��

��

���������

��

��

��

��

�������

�������

�������

���

most challenging dataset, DomainNet, which contains more classes and more complex domain shifts. Results in Table 4 show the significant performance of the proposed SIG method, which provides 3 . 3% averaged promotion, although the performance in the task of Sketch is slightly lower than that of DRT+ST. Compared with iMSDA, our SIG overpasses by a large margin under a more general data generation process.

Ablation Study: To evaluate the effectiveness of individual loss terms, we also devise the two model variants. 1) SIG-sem : remove the class-aware alignment loss. 2) SIG-vae : remove the reconstruction loss and the KL divergence loss. Experiment results on the Office-Home dataset are shown in Figure 4. We can find that the class-aware alignment loss plays an important role in the model performance, reflecting that the class-aware alignment can mitigate the influence of target shift. We also discover that incorporating the reconstruction and KL divergence has a positive impact on the overall performance of the model, which shows the necessity of modeling the marginal distributions.

6 Conclusion

This paper presents a general data generation process for multi-source domain adaptation, which coincides with real-world scenarios. Based on this data generation process, we prove that the changing latent variables are subspace identifiable, which provides a novel solution for disentangled representation. Compared with the existing methods, the proposed subspace identification theory requires fewer auxiliary variables and frees the model from the monotonic transformation of latent variables, making it possible to apply the proposed method to real-world data. Experiment results on several mainstream benchmark datasets further evaluate the effectiveness of the proposed subspace identification guaranteed model. In summary, this paper takes a meaningful step for causal representation learning. Broader Impacts: SIG disentangles the latent variables to create a model that is more transparent, thereby aiding in the reduction of bias and the promotion of fairness. Limitation: However, the proposed subspace identification still requires several assumptions that might not be met in real-world scenarios. Therefore, how further to relax the assumptions, i.e., conditional independent assumption, would be an interesting future direction.

References

Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734, 2019.

Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems, 29, 2016.

Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In IJCAI: proceedings of the conference, volume 2019, page 2060. NIH Public Access, 2019.

Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domainspecific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 7354-7362, 2019.

Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3296-3303, 2019.

Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 627-636, 2019.

Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pages 1081-1090. PMLR, 2019.

Yuansi Chen and Peter Bühlmann. Domain adaptation under structural causal models. The Journal of Machine Learning Research, 22(1):11856-11935, 2021.

Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287314, 1994.

Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16654-16663, 2021.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.

Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.

Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pages 2839-2848. PMLR, 2016.

Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-948. PMLR, 2020.

Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34:1624-1633, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. Pmlr, 2018.

Aapo Hyvärinen. Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1984):20110534, 2013.

Aapo Hyvarinen, Juha Karhunen, and Erkki Oja. Independent component analysis. Studies in informatics and control, 11(2):205-207, 2002.

Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. arXiv preprint arXiv:2302.02672, 2023.

Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.

Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.

Aapo Hyv"arinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.

Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.

Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 464-480. Springer, 2020.

Guoliang Kang, Lu Jiang, Yunchao Wei, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for single-and multi-source domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 44(4):1793-1804, 2020.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.

Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33:12768-12778, 2020.

David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10930, 2020.

Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pages 11455-11472. PMLR, 2022.

Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848, 2017.

Da Li and Timothy Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI, pages 382-403. Springer, 2020.

Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9991-10000, 2021.

Yunsheng Li, Lu Yuan, Yinpeng Chen, Pei Wang, and Nuno Vasconcelos. Dynamic transfer for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10998-11007, 2021.

Zhenpeng Li, Zhen Zhao, Yuhong Guo, Haifeng Shen, and Jieping Ye. Mutual learning network for multi-source domain adaptation. arXiv preprint arXiv:2003.12944, 2020.

Zijian Li, Ruichu Cai, Tom ZJ Fu, and Kun Zhang. Transferable time-series forecasting under causal conditional shift. arXiv preprint arXiv:2111.03422, 2021.

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41144124. PMLR, 2019.

Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258, 2019.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.

Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208-2217. PMLR, 2017.

Sara Magliacane, Thijs Van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems, 31, 2018.

Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Boosting domain adaptation by discovering latent domains. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3771-3780, 2018.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359, 2010.

Geon Yeong Park and Sang Wan Lee. Information-theoretic regularization for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9214-9223, 2021.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.

Chuan-Xian Ren, Yong-Hui Liu, Xi-Wen Zhang, and Ke-Kun Huang. Multi-source unsupervised domain adaptation via pseudo target domain. IEEE Transactions on Image Processing, 31:21222135, 2022.

Manley Roberts, Pranav Mani, Saurabh Garg, and Zachary Lipton. Unsupervised learning under latent label shift. Advances in Neural Information Processing Systems, 35:18763-18778, 2022.

Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210-20229, 2021.

Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.

Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.08735, 2018.

Changjian Shui, Zijian Li, Jiaqi Li, Christian Gagné, Charles X Ling, and Boyu Wang. Aggregating from multiple target-shifted sources. In International Conference on Machine Learning, pages 9638-9648. PMLR, 2021.

Petar Stojanov, Mingming Gong, Jaime Carbonell, and Kun Zhang. Data-driven approach to multiple-source domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3487-3496. PMLR, 2019.

Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations to learn? Advances in Neural Information Processing Systems, 34:24791-24803, 2021.

Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.

Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33:19276-19289, 2020.

Takeshi Teshima, Issei Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mechanism transfer. In International Conference on Machine Learning, pages 9458-9469. PMLR, 2020.

Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pages 10401-10412. PMLR, 2021.

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.

Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 727-744. Springer, 2020.

Zengmao Wang, Chaoyang Zhou, Bo Du, and Fengxiang He. Self-paced supervision for multisource domain adaptation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022.

Jun Wen, Nenggan Zheng, Junsong Yuan, Zhefeng Gong, and Changyou Chen. Bayesian uncertainty matching for unsupervised domain adaptation. arXiv preprint arXiv:1906.09693, 2019.

Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi-source domain adaptation. In International Conference on Machine Learning, pages 10214-10224. PMLR, 2020.

Shaoan Xie, Lingjing Kong, Mingming Gong, and Kun Zhang. Multi-domain image generation and translation with identifiability guarantees. In The Eleventh International Conference on Learning Representations.

Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning, pages 5423-5432. PMLR, 2018.

Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3964-3973, 2018.

Luyu Yang, Yogesh Balaji, Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 608-624. Springer, 2020.

Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. arXiv preprint arXiv:2210.13647, 2022.

Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428, 2021.

Kun Zhang and Laiwan Chan. Kernel-based nonlinear independent component analysis. In Independent Component Analysis and Signal Separation: 7th International Conference, ICA 2007, London, UK, September 9-12, 2007. Proceedings 7, pages 301-308. Springer, 2007.

Kun Zhang and Laiwan Chan. Minimal nonlinear distortion principle for nonlinear independent component analysis. Journal of Machine Learning Research, 9(Nov):2455-2487, 2008.

Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.

Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, Qingsong Liu, and Clark Glymour. Domain adaptation as a problem of inference on graphical models. Advances in neural information processing systems, 33:4965-4976, 2020.

Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference on machine learning, pages 819-827. PMLR, 2013.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International conference on machine learning, pages 7404-7413. PMLR, 2019.

Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.

Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica with unconditional priors. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.

Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5989-5996, 2019.

Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE transactions on neural networks and learning systems, 32(4):1713-1722, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered multiple-source domain adaptation problems in the context of causal representation learning. Specifically, this paper considered quite general scenarios in the domain adaptation by figuring four different representations.\n\n- Domain specific and label irrelevant \n- Domain specific and label relevant\n- Domain invariant and label irrelevant \n- Domain invariant and label relevant\n\nNotably, this paper derived theoretical assumptions via illustrating when we could identify these four representations (or subspace). A VAE based approach in deep learning is further proposed to identify the subspace. Empirical results are further validated in synthetic data, benchmark domain adaptation dataset (including challenging DomainNET dataset)\n\nSTRENGTHS:\nOverall this reviewer feels that this paper has a solid contribution in the theoretical aspects in causal based domain adaptation. Extensive experiments further validated the practical utility. \n\n- **Significance & Originality**\nThis paper proposes a very solid theoretical justification in the identifiability issue in multi-source da, which is a fundamental problem. More importantly, this paper studied the general setting (or most difficult settings) in multi-source da. Besides, theoretical assumptions are proposed, which is generally non-trivial.\n\n- **Quality**\nThe theoretical assumptions and proof seem valid and reasonable for me. The proposed practical method is also reasonable. Empirical results in toy data and real-data demonstrated that the proposed method could better identify different representations. \n\n- **Clarity**\nIn general, this paper is clearly written and well-explained. Since this paper is a bit theoretical, there are still several theoretical points that need to be clarified (see the weakness part).\n\nWEAKNESSES:\nSince this paper covers several fundamental proofs, this reviewer feels uncertain about several points within the paper. \n\n\n1. Theorem 1 about theoretical assumptions. In general, I could follow the proof, where intuitions within these assumptions could be better explained. I just take theorem 1 as an example.\n\n- About conditional independence. If I understand correctly, this implies, the representation variable $P(Z|U) = \\prod P(z_i|U)$ ?  (sort of disentanglement or mean-field approximation assumption)? If we write it as a log probability term, it will be $\\log P(Z|U) = \\sum \\log P(Z_i|U)$, which is consistent with your paper…\n- About linear independence. This assumption seems to illustrate that the gradient of the probability density over each component should be linearly independent? That implies if we use a gradient based approach to learn each component, each component will converge to a distinct direction. Thus we could possibly identify different independent components?\n- I have a general question in theory 1 (for example). Is it possible to extend this assumption into the blockwise vectors? i.e, $P(Z|U) = P(Z_1|U)P(Z_2|U)P(Z_3|U)P(Z_4|U)$, where $Z_1$ is a subspace vector rather than a scalar. \n\n2. About explanations in toy data. It seems that the proposed method works quite significantly in toy data. However the experimental details (such as the data generation) could be better clarified. For example, is there some sort of spurious correlation?\n\n3. In Equation (1) Line 3, I would think this holds when we have figure 1 by considering the conditional independence. \n\n4. I like the example in explaining different subspace in line 82-92. It could be better to consider this example into more real-world scenarios such as drug discovery and health. For example, data collected from different hospitals may create different $Z_1$, this could be helpful to better understand the importance of considering a general scenario.\n\nQUESTIONS:\nSee comments in the weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of multi source domain adaptation where we have access to multiple labeled source domains and unlabeled target domain. Authors consider a novel data generation process by modeling them through 4 new latent variables (i.e. combinations of domain specific or domain invariant and label relevant/ label irrelevant). This is done to relax the assumptions made by previous theoretical analyses. The authors provide sufficient theoretical analysis in this context by providing subspace identification guarantees and using that to improve the results. They conduct multiple experiments include synthetic experiments and the results are satisfactory.\n\nSTRENGTHS:\n*  The paper is well-presented. \n* The paper provides new framework through their modeling of data generation process by introducing the 4 latent variables and then considering them to influences from the domain index and the target labels. \n* Their theoretical analysis is convincing and experiments are impressive.\n\nWEAKNESSES:\n* This framework is somewhat similar to an earlier work on domain generalization(https://arxiv.org/pdf/2102.11436.pdf) . I agree that your data generation process further expands on their structural causal model but noting the similarities it should be cited and expand on the differences\n* Importantly, since we model label relevant attributes separately, should we still make the assumption that source and target contain the same labels. \n* My question specifically is can you comment on if this framework naturally support open-set/partial MSDAs.\n\nQUESTIONS:\nPlease comment on the weakness I raised above and if you could provide some results even on one dataset where there is either a lesser number of classes in target than in source (https://arxiv.org/abs/1808.04205)  or like in other open set DA problems that would be great\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigate the problem of multi-source domain adaptation. In detail, the authors first devise a causal generation process, which consider the domain-specific and label-irrelevant, domain-specific and label-relevant, domain-invariant and label-invariant, domain-invariant and label-irrelevant variables. Based on the aforementioned generation process, the authors raise the identification guarantees for the latent variables. The authors also evaluate the proposed methods on several benchmarks and achieve good performance.\n\nSTRENGTHS:\n1.\tThe authors provide an interesting perspective for multi-source domain adaptation, which uses nonlinear ICA to identify the latent variables. Compared with the existing methods, the proposed method relaxes the assumptions.\n2.\tThe authors evaluate the proposed methods on several datasets and achieve the ideal results.\n\nWEAKNESSES:\n1.\tAccording to the theoretical results, the authors give three assumptions which is similar to those of nonlinear ICA literature. Why are these existing assumptions applicable in real-world scenarios? Moreover, according to the subspace identification results, the dimension of latent variables is small and restricted by the number of domains, which is hard to be met in practice.    \n2.\tFor the real-world scenario, the number of ground-truth latent variables is unknown, is it reasonable to assume that their number is according to the observed domains?    \n3.\tThe authors equip a cross-attention module to the ResNet101 for the DomainNet dataset, it is suggested the authors provide the experiment results of other datasets, e.g., Office-home, with the proposed cross-attention module.    \n\nMinors:     \nThere are some typos in this paper. For example, the illustrations of Lemma2 in the main and appendix are inconsistent.\n\nQUESTIONS:\nNO\n\nLIMITATIONS:\nNON", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework to tackle the problem of multi-source unsupervised domain adaptation. Existing methods often have stringent requirements, such as a large number of domains and invariant label distributions. To address these limitations, this paper proposes a subspace identification theory that disentangles domain-invariant and domain-specific variables under less restrictive constraints. The proposed Subspace Identification Guarantee (SIG) model is based on variational inference and additionally incorporates class-aware conditional alignment to handle domain shifts. Experimental results demonstrate that the SIG model outperforms existing SOTA techniques on four benchmark datasets (OfficeHome, ImageCLEF, PACS, DomainNet), showcasing its effectiveness in real-world applications.\n\nSTRENGTHS:\n- The paper defines three major drawbacks of [a]  – invariant label distribution, requiring a large number of domains and monotonic transformation between latent variables. These assumptions are relaxed by the *Subspace Identification Guarantee* (SIG) model proposed in this paper. Most interestingly, while Kong et al. claimed that for $n$ dimensional latent space, $2n+1$ domains would be necessary, authors here have shown that “n+1” domains are sufficient. \n- The core ideas on subspace identifiability are theoretically grounded. The paper defines a new data generative process based on a 4-way split of the latent space and builds upon the theoretical results to define a new multi-source unsupervised domain adaptation framework using a VAE-based architecture.\n- Results are presented on 4 benchmark datasets, with the proposed model outperforming all other SOTA approaches.\n\n[a] Kong et. al., Partial Identifiability for Domain Adaptation, ICML 2022\n\nWEAKNESSES:\n- In the data generation process, the paper introduces 4 latent variables from the product of two sets {domain-specific, domain-invariant} $\\times$ {label-relevant, label-irrelevant}. I am unsure what the intuitive reasoning behind the \"domain-invariant label-irrelevant\" latent variable $\\mathbf{z}_4$ is. \n- There are no individual constraints on $\\mathbf{z}_1$ and $\\mathbf{z}_4$ during the learning process. Are they identifiable individually?\n- While the paper claims that the proposed SIG model can handle all three types of shifts: covariate, target, and conditional, experiments focus primarily on covariate shift (based on the datasets used). It’s unclear how the datasets, which contain mostly covariate shifts, were used for studying the two other shifts.\n- The clarity could be improved, especially when introducing the SIG model. For example, in Section 4.2, $\\hat{\\mathbf{z}}_{3,\\mathcal{S}}^{(i)}$ is denoted as \"..latent variables of i-th class from source\". How is this estimated?\n\nQUESTIONS:\nPlease see weaknesses. Also, the derivation of the ELBO in Equation 5 should be provided to enhance the completeness of the paper.\n\nLIMITATIONS:\nThe authors discuss the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}