SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zijian Li $^{2,3}$ , Ruichu Cai $^{2*}$ , Guangyi Chen $^{3,1}$ , Boyang Sun $^{3}$ , Zhifeng Hao $^{4}$ , Kun Zhang $^{3,1*}$
$^{2}$  School of Computer Science, Guangdong University of Technology
<sup>3</sup> Mohamed bin Zayed University of Artificial Intelligence
4 Shantou University
Multi-source domain adaptation (MSDA) methods aim to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Although current methods achieve target joint distribution identifiability by enforcing minimal changes across domains, they often necessitate stringent conditions, such as an adequate number of domains, monotonic transformation of latent variables, and invariant label distributions. These requirements are challenging to satisfy in real-world applications. To mitigate the need for these strict assumptions, we propose a subspace identification theory that guarantees the disentanglement of domain-invariant and domain-specific variables under less restrictive constraints regarding domain numbers and transformation properties, thereby facilitating domain adaptation by minimizing the impact of domain shifts on invariant variables. Based on this theory, we develop a Subspace Identification Guarantee (SIG) model that leverages variational inference. Furthermore, the SIG model incorporates class-aware conditional alignment to accommodate target shifts where label distributions change with the domains. Experimental results demonstrate that our SIG model outperforms existing MSDA techniques on various benchmark datasets, highlighting its effectiveness in real-world applications.
Multi-Source Domain Adaptation (MSDA) is a method of transferring knowledge from multiple labeled source domains to an unlabeled target domain, to address the challenge of domain shift between the training data and the test environment. Mathematically, in the context of MSDA, we assume the existence of  $M$  source domains  $\{S_1,S_2,\dots,S_M\}$  and a single target domain  $\mathcal{T}$ . For each source domain  $S_{i}$ , data are drawn from a distinct distribution, represented as  $p_{\mathbf{x},\mathbf{y}|\mathbf{u}_{S_i}}$ , where the variables  $\mathbf{x},\mathbf{y},\mathbf{u}$  correspond to features, labels, and domain indices, respectively. In a similar manner, the distribution within the target domain  $\mathcal{T}$  is given by  $p_{\mathbf{x},\mathbf{y}|\mathbf{u}_{\mathcal{T}}}$ . In the source domains, we have access to  $m_{i}$  annotated feature-label pairs of each domain, denoted by  $(\mathbf{x}^{\mathcal{S}_i},\mathbf{y}^{\mathcal{S}_i}) = (\mathbf{x}_k^{\mathcal{S}_i},y_k^{\mathcal{S}_i})_{k = 1}^{m_i}$ , while in the target domain, only  $m_{\mathcal{T}}$  unannotated features are observed, represented as  $(\mathbf{x}^{(\mathcal{T})}) = (\mathbf{x}_k^{\mathcal{T}})_{k = 1}^{m_{\mathcal{T}}}$ . The primary goal of MSDA is to effectively leverage these labeled source data and unlabeled target data to identify the target joint distribution  $p_{\mathbf{x},\mathbf{y}|\mathbf{u}_{\mathcal{T}}}$ .
However, identifying the target joint distribution of  $\mathbf{x}$ ,  $\mathbf{y}|\mathbf{u}_{\mathcal{T}}$  using only  $\mathbf{x}|\mathbf{u}_{\mathcal{T}}$  as observations present a significant challenge, since the possible mappings from  $p_{\mathbf{x},\mathbf{y}|\mathbf{u}_{\mathcal{T}}}$  to  $p_{\mathbf{x}|\mathbf{u}_{\mathcal{T}}}$  are infinite when no extra constraints are given. To solve this problem, some assumptions have been proposed to constrain the domain shift, such as covariate shift [44], target shift [58, 1], and conditional shift [3, 56]. For example, the most conventional covariate shift assumption posits that  $p_{\mathbf{y}|\mathbf{x}}$  is fixed across different domains
while  $p_{\mathbf{x}}$  varies. Under this assumption, researchers can employ techniques such as importance reweighting [52], invariant representation learning [11], or cycle consistency [17] for distribution alignment. Additionally, target shift assumes that  $p_{\mathbf{x}|\mathbf{y}}$  is fixed while the label distribution  $p_{\mathbf{y}}$  changes, whereas conditional shift is represented by a fixed  $p_{\mathbf{y}}$  and a varying  $p_{\mathbf{x}|\mathbf{y}}$ . More generally, the minimal change principle has been proposed, which not only unifies the aforementioned assumptions but also enables the theoretical guarantee of the identifiability of the target joint distribution. Specifically, it assumes that  $p_{\mathbf{y}|\mathbf{u}}$  and  $p_{\mathbf{x}|\mathbf{y},\mathbf{u}}$  change independently and the change of  $p_{\mathbf{x}|\mathbf{y},\mathbf{u}}$  is minimal. Please refer to Appendix G for further discussion of related works, including domain adaptation and identification.
Although current methods demonstrate the identifiability of the target joint distribution through the minimal change principle, they often impose strict conditions on the data generation process and the number of domains, limiting their practical applicability. For instance, iMSDA [30] presents the component-wise identification of the domain-changed latent variables, subsequently identifying target joint distribution by modeling a data generation process with variational inference. However, this identification requires the following conditions. First, a sufficient number of auxiliary variables is employed for the component-wise theoretical guarantees, meaning that when the dimension of latent variables is  $n$ , a total of  $2n + 1$  domains are needed. Second, in order to identify high-level invariant latent variables, a component-wise monotonic function between latent variables must be assumed. Third, these methods implicitly assume that label distribution remains stable across domains, despite the prevalence of target shift in real-world scenarios. These conditions are often too restrictive to be met in practice, highlighting the need for a more general approach to identifying latent variables across a wider range of domain shifts.
In an effort to alleviate the need for such strict assumptions, we present a subspace identification theory in this paper that guarantees the disentanglement of domain-invariant and domain-specific variables under more relaxed constraints concerning the number of domains and transformation properties. In contrast to component-wise identification, our subspace identification method demands fewer auxiliary variables (i.e., when the dimension of latent variables is  $n$ , only  $n + 1$  domains are required). Additionally, we design a more general data generation process that accounts for target shift and does not necessitate monotonic transformation between latent variables. In this process, we categorize latent variables into four groups based on whether they are influenced by domain index or label. Building on the theory and causal generation process, we develop a Subspace Identification Guarantee (SIG) model that employs variational inference to identify latent variables. A class-aware condition alignment is incorporated to mitigate the impact of target shift, ensuring the update of the most confident cluster embedding. Our approach is validated through a simulation experiment for subspace identification evaluation and four widely-used public domain adaptation benchmarks for application evaluation. The impressive performance that outperforms state-of-the-art methods demonstrates the effectiveness of our method.
2 Identifying Target Joint Distribution with Data Generation Process
2.1 Data Generation Process
We begin with introducing the data generation process. As shown in Figure 1, the observed data  $\mathbf{x} \in \mathcal{X}$  are generated by latent variables  $\mathbf{z} \in \mathcal{Z} \subseteq \mathbb{R}^n$ . Sequentially, we divide the latent variables  $\mathbf{z}$  into the four parts, i.e.  $\mathbf{z} = \{\mathbf{z}_1, \mathbf{z}_2, \mathbf{z}_3, \mathbf{z}_4\} \in \mathcal{Z} \subseteq \mathbb{R}^n$ , which are shown as follows.
- domain-specific and label-irrelevant variables  $\mathbf{z}_1 \in \mathbb{R}^{n_1}$ .
- domain-specific but label-relevant variables  $\mathbf{z}_2 \in \mathbb{R}^{n_2}$ .
- domain-invariant and label-relevant variables  $\mathbf{z}_3\in \mathbb{R}^{n_3}$
- domain-invariant but label-irrelevant variables  $\mathbf{z}_4 \in \mathbb{R}^{n_4}$ .
To better understand these latent variables, we provide some examples in DomainNet datasets. First,  $\mathbf{z}_1 \in \mathbb{R}^{n_1}$  denotes the styles of the images like "infograph" and "sketch", which are irrelevant to labels.  $\mathbf{z}_2 \in \mathbb{R}^{n_2}$  denotes the latent variables that can be the texture information relevant to domains and labels.
For example, the samples of "clock" and "telephone" contain some digits, and these digits in these samples are a special texture, which can be used for classification and be influenced by different
Figure 1: Data generation process, where the gray the white nodes denote the observed and latent variables, respectively.
styles, such as "infograph".  $\mathbf{z}_3\in \mathbb{R}^{n_3}$  denotes the latent variables that are only relevant to the labels. For example, in the DomainNet dataset, it can be interpreted as the meaning of different classes like "Bicycle" or "Teapot". Finally,  $\mathbf{z}_4\in \mathbb{R}^{n_4}$  denotes the label-irrelevant latent variables. For example,  $\mathbf{z}_4$  can be interpreted as the background that is invariant to domains and labels.
Based on the definitions of these latent variables, we let the observed data be generated from  $\mathbf{z}$  through an invertible and smooth mixing function  $g: \mathcal{Z} \to \mathcal{X}$ . Due to the target shift, we further consider that the  $p_{\mathbf{y}}$  is influenced by  $\mathbf{u}$ , i.e.  $\mathbf{u} \to \mathbf{y}$ .
Compared with the existing data generation process like [30], the proposed data generation process is different in three folds. First,  $p_{\mathbf{u}}$  is independent of  $p_{\mathbf{y}}$  in the iMSDA [30], so the target shift is not taken into account. Second, the data generation process of iMSDA requires an invertible and monotonic function between latent variables for component-wise identification, which is too strict to be met in practice. Third, to provide a more general way to depict the real-world data, our data generation process introduces the domain-specific but label-relevant latent variables  $\mathbf{z}_{s_2}$  and the domain-invariant but label-irrelevant variables  $\mathbf{z}_4$ .
2.2 Identifying the Target Joint Distribution
In this part, we show how to identify the target joint distribution  $p_{\mathbf{x},\mathbf{y}|\mathbf{u}_{\mathcal{T}}}$  with the help of marginal distribution. By introducing the latent variables and combining the proposed data generation process, we can obtain the following derivation.
$$
\begin{array}{l} p _ {\mathbf {x}, \mathbf {y} | \mathbf {u} _ {\mathcal {T}}} = \int_ {\mathbf {z} _ {1}} \int_ {\mathbf {z} _ {2}} \int_ {\mathbf {z} _ {3}} \int_ {\mathbf {z} _ {4}} p _ {\mathbf {x}, \mathbf {y}, \mathbf {z} _ {1}, \mathbf {z} _ {2}, \mathbf {z} _ {3}, \mathbf {z} _ {4} | \mathbf {u} _ {\mathcal {T}}} d \mathbf {z} _ {1} d \mathbf {z} _ {2} d \mathbf {z} _ {3} d \mathbf {z} _ {4} \\ = \int_ {\mathbf {z} _ {1}} \int_ {\mathbf {z} _ {2}} \int_ {\mathbf {z} _ {3}} \int_ {\mathbf {z} _ {4}} p _ {\mathbf {x}, \mathbf {z} _ {1}, \mathbf {z} _ {2}, \mathbf {z} _ {3}, \mathbf {z} _ {4} | \mathbf {y}, \mathbf {u} _ {\mathcal {T}}} \cdot p _ {\mathbf {y} | \mathbf {u} _ {\mathcal {T}}} d \mathbf {z} _ {1} d \mathbf {z} _ {2} d \mathbf {z} _ {3} d \mathbf {z} _ {4} \tag {1} \\ = \int_ {\mathbf {z} _ {1}} \int_ {\mathbf {z} _ {2}} \int_ {\mathbf {z} _ {3}} \int_ {\mathbf {z} _ {4}} p _ {\mathbf {x} | \mathbf {z} _ {1}, \mathbf {z} _ {2}, \mathbf {z} _ {3}, \mathbf {z} _ {4}} \cdot p _ {\mathbf {z} _ {1}, \mathbf {z} _ {2}, \mathbf {z} _ {3}, \mathbf {z} _ {4} | \mathbf {y}, \mathbf {u} _ {\mathcal {T}}} \cdot p _ {\mathbf {y} | \mathbf {u} _ {\mathcal {T}}} d \mathbf {z} _ {1} d \mathbf {z} _ {2} d \mathbf {z} _ {3} d \mathbf {z} _ {4}. \\ \end{array}
$$
According to the derivation in Equation (1), we can identify the target joint distribution by modeling three distributions. First, we need to model  $p_{\mathbf{x}|\mathbf{z}_1,\mathbf{z}_2,\mathbf{z}_3,\mathbf{z}_4}$ , implying that we need to model the conditional distribution of observed data give latent variables, which coincides with a generative model for observed data. Second, we need to estimate the label pseudo distribution of target domain  $p_{\mathbf{y}|\mathbf{u}_{\mathcal{T}}}$ . Third, we need to model  $p_{\mathbf{z}_1,\mathbf{z}_2,\mathbf{z}_3,\mathbf{z}_4|\mathbf{y},\mathbf{u}_{\mathcal{T}}}$  meaning that the latent variables should be identified with auxiliary variables  $\mathbf{u}$ ,  $\mathbf{y}$  under theoretical guarantees. In the next section, we will introduce how to identify these latent variables with subspace identification block-wise identification results.
3 Subspace Identifiability for Latent Variables
In this section, we provide how to identify the latent variables in Figure 1. In detail, we first prove that  $\mathbf{z}_2$  is subspace identifiable and  $\mathbf{z}_1, \mathbf{z}_3$  can be reconstructed from the estimated  $\hat{\mathbf{z}}_1, \hat{\mathbf{z}}_2, \hat{\mathbf{z}}_3$ . Then we further prove that  $\mathbf{z}_4$  is block-wise identifiable.
To clearly introduce the subspace identification theory, we employ a simple data generation process [3] as shown in Figure 2. In this data generation process,  $\mathbf{z}_s \in \mathbb{R}^{n_s}$  and  $\mathbf{z}_c \in \mathbb{R}^{n_c}$  denote the domain-specific and domain-invariant latent variables, respectively. For convenient, we let  $\mathbf{z} = \{\mathbf{z}_s, \mathbf{z}_c\}, n = n_s + n_c$ . Moreover, we assume  $\mathbf{z}_s = (z_i)_{i=1}^{n_s}$  and  $\mathbf{z}_c = (z_i)_{i=n_s+1}^n$ . And  $\{\mathbf{u}, \mathbf{y}, \mathbf{x}\}$  denote the domain index, labels, and observed data, respectively. And we further let the observed data be generated from  $\mathbf{z}$  through an invertible and smooth mixing function  $g: \mathcal{Z} \to \mathcal{X}$ . The subspace identification of  $\mathbf{z}_s$  means that for each ground-truth  $z_{s,i}$ , there exists  $\hat{\mathbf{z}}_s$  and an invertible function  $h_i: \mathbb{R}^n \to \mathbb{R}$ , such that  $z_{s,i} = h_i(\hat{\mathbf{z}}_s)$ .
Figure 2: A simple data generalization process for introducing subspace identification.
Theorem 1. (Subspace Identification of  $\mathbf{z}_s$ .) We follow the data generation process in Figure 2 and make the following assumptions:
- A1 (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e.,  $p_{\mathbf{z}|\mathbf{u}} > 0$  over  $\mathcal{Z}$  and  $\mathcal{U}$ .
- A2 (Conditional independent): Conditioned on  $\mathbf{u}$ , each  $z_{i}$  is independent of any other  $z_{j}$  for  $i, j \in \{1, \dots, n\}, i \neq j$ , i.e.  $\log p_{\mathbf{z}|\mathbf{u}}(\mathbf{z}|\mathbf{u}) = \sum_{i}^{n} q_{i}(z_{i}, \mathbf{u})$  where  $q_{i}(z_{i}, \mathbf{u})$  is the log density of the conditional distribution, i.e.,  $q_{i} : \log p_{z_{i}|\mathbf{u}}$ .
- A3 (Linear independence): For any  $\mathbf{z}_s \in \mathcal{Z}_s \subseteq \mathbb{R}^{n_s}$ , there exist  $n_s + 1$  values of  $\mathbf{u}$ , i.e.,  $\mathbf{u}_j$  with  $j = 0, 1, \dots, n_s$ , such that these  $n_s$  vectors  $\mathbf{w}(\mathbf{z}, \mathbf{u}_j) - \mathbf{w}(\mathbf{z}, \mathbf{u}_0)$  with  $j = 1, \dots, n_s$  are linearly independent, where vector  $\mathbf{w}(\mathbf{z}, \mathbf{u}_j)$  is defined as follows:
$$
\mathbf {w} (\mathbf {z}, \mathbf {u}) = \left(\frac {\partial q _ {1} \left(z _ {1} , \mathbf {u}\right)}{\partial z _ {1}}, \dots , \frac {\partial q _ {i} \left(z _ {i} , \mathbf {u}\right)}{\partial z _ {i}}, \dots \frac {\partial q _ {n _ {s}} \left(z _ {n _ {s}} , \mathbf {u}\right)}{\partial z _ {n _ {s}}}\right), \tag {2}
$$
By modeling the aforementioned data generation process,  $\mathbf{z}_s$  is subspace identifiable.
Proof sketch. First, we construct an invertible transformation  $h$  between the ground-truth  $\mathbf{z}$  and estimated  $\hat{\mathbf{z}}$ . Sequentially, we leverage the variance of different domains to construct a full-rank linear system, where the only solution of  $\frac{\partial \mathbf{z}_s}{\partial \hat{\mathbf{z}}_c}$  is zero. Since the Jacobian of  $h$  is invertible, for each  $z_{s,i}, i \in \{1, \dots, n_s\}$ , there exists a  $h_i$  such that  $z_{s,i} = h_i(\hat{\mathbf{z}})$  and  $\mathbf{z}_s$  is subspace identifiable.
The proof can be found in Appendix B.1. The first two assumptions are standard in the component-wise identification of existing nonlinear ICA [30, 27]. The third Assumption means that  $p_{\mathbf{z}|\mathbf{u}}$  should vary sufficiently over  $n + 1$  domains. Compared to component-wise identification, which necessitates  $2n + 1$  domains and is likely challenging to fulfill, subspace identification can yield equivalent results in terms of identifying the ground-truth latent variables with only  $n + 1$  domains. Therefore, subspace identification benefits from a more relaxed assumption.
Based on the theoretical results of subspace identification, we show that the ground-truth  $\mathbf{z}_1, \mathbf{z}_2$  and  $\mathbf{z}_3$  be reconstructed from the estimated  $\hat{\mathbf{z}}_1, \hat{\mathbf{z}}_2$  and  $\hat{\mathbf{z}}_3$ . For ease of exposition, we assume that  $\mathbf{z}_1, \mathbf{z}_2, \mathbf{z}_3$ , and  $\mathbf{z}_4$  correspond to components in  $\mathbf{z}$  with indices  $\{1, \dots, n_1\}, \{n_1 + 1, \dots, n_1 + n_2\}, \{n_1 + n_2 + 1, \dots, n_1 + n_2 + n_3\}$ , and  $\{n_1 + n_2 + n_3 + 1, \dots, n\}$ , respectively.
Corollary 1.1. We follow the data generation in Section 2.1, and make the following assumptions which are similar to A1-A3:
A4 (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e.,  $p_{\mathbf{z}|\mathbf{u},\mathbf{y}} > 0$  over  $\mathcal{Z}$ ,  $\mathcal{U}$ , and  $\mathcal{V}$ .
A5 (Conditional independent): Conditioned on  $\mathbf{u}$  and  $\mathbf{y}$ , each  $z_{i}$  is independent of any other  $z_{j}$  for  $i,j\in \{1,\dots ,n\} ,i\neq j$  i.e.  $\log p_{\mathbf{z}|\mathbf{u},\mathbf{y}}(\mathbf{z}|\mathbf{u},\mathbf{y}) = \sum_{i}^{n}q_{i}(z_{i},\mathbf{u},\mathbf{y})$  where  $q_{i}(z_{i},\mathbf{u},\mathbf{y})$  is the log density of the conditional distribution, i.e.,  $q_{i}:\log p_{z_{i}|\mathbf{u},\mathbf{y}}$ .
A6 (Linear independence): For any  $\mathbf{z} \in \mathcal{Z} \subseteq \mathbb{R}^n$ , there exists  $n_1 + n_2 + n_3 + 1$  combination of  $(\mathbf{u}, \mathbf{y})$ , i.e.  $j = 1, \dots, U$  and  $c = 1, \dots, C$  and  $U \times C - 1 = n_1 + n_2 + n_3$ , where  $U$  and  $C$  denote the number of domains and the number of labels. Such that these  $n_1 + n_2 + n_3$  vectors  $\mathbf{w}(\mathbf{z}, \mathbf{u}_j, \mathbf{y}_c) - \mathbf{w}(\mathbf{z}, \mathbf{u}_0, \mathbf{y}_0)$  are linearly independent, where  $\mathbf{w}(\mathbf{z}, \mathbf{u}_j, \mathbf{y}_c)$  is defined as follows:
$$
\mathbf {w} (\mathbf {z}, \mathbf {u}, \mathbf {y}) = \left(\frac {\partial q _ {1} \left(z _ {1} , \mathbf {u} , \mathbf {y}\right)}{\partial z _ {1}}, \dots , \frac {\partial q _ {i} \left(z _ {i} , \mathbf {u} , \mathbf {y}\right)}{\partial z _ {i}}, \dots \frac {\partial q _ {n} \left(z _ {n} , \mathbf {u} , \mathbf {y}\right)}{\partial z _ {n}}\right). \tag {3}
$$
By modeling the data generation process in Section 2.1,  $\mathbf{z}_2$  is subspace identifiable, and  $\mathbf{z}_1, \mathbf{z}_3$  can be reconstructed from  $\hat{\mathbf{z}}_1, \hat{\mathbf{z}}_2$  and  $\hat{\mathbf{z}}_2, \hat{\mathbf{z}}_3$ , respectively.
Proof sketch. The detailed proof can be found in Appendix B.2. First, we construct an invertible transformation  $h$  to bridge the relation between the ground-truth  $\mathbf{z}$  and the estimated  $\hat{\mathbf{z}}$ . Then, we repeatedly use Theorem 1 three times by considering the changing of labels and domains. Hence, we find that the values of some blocks of the Jacobian of  $h$  are zero. Finally, the Jacobian of  $h$  can be formalized as Equation (4).
where  $J_{h}$  denotes the Jacobian of  $h$  and  $\pmb{J}_{h}^{ij} := \frac{\partial\mathbf{z}_{i}}{\partial\hat{\mathbf{z}}_{j}}$  and  $i,j\in \{1,2,3,4\}$ . Since  $h(\cdot)$  is invertible,  $\pmb{J}_h$  is a full-rank matrix. Therefore, for each  $z_{2,i},i\in \{n_1 + 1,\dots ,n_1 + n_2\}$ , there exists a  $h_{2,i}$  such that  $z_{2,i} = h_i(\hat{\mathbf{z}}_2)$ . Moreover, for each  $z_{1,i},i\in \{1,\dots ,n_1 + 1\}$ , there exists a
$$
\boldsymbol {J} _ {h} = \left[ \begin{array}{c c c c} \boldsymbol {J} _ {h} ^ {1, 1} & \boldsymbol {J} _ {h} ^ {1, 2} & \boldsymbol {J} _ {h} ^ {1, 3} = 0 & \boldsymbol {J} _ {h} ^ {1, 4} = 0 \\ \hline \boldsymbol {J} _ {h} ^ {2, 1} = 0 & \boldsymbol {J} _ {h} ^ {2, 2} & \boldsymbol {J} _ {h} ^ {2, 3} = 0 & \boldsymbol {J} _ {h} ^ {2, 4} = 0 \\ \hline \boldsymbol {J} _ {h} ^ {3, 1} = 0 & \boldsymbol {J} _ {h} ^ {3, 2} & \boldsymbol {J} _ {h} ^ {3, 3} & \boldsymbol {J} _ {h} ^ {3, 4} = 0 \\ \hline \boldsymbol {J} _ {h} ^ {4, 1} & \boldsymbol {J} _ {h} ^ {4, 2} & \boldsymbol {J} _ {h} ^ {4, 3} & \boldsymbol {J} _ {h} ^ {4, 4} \end{array} \right], \tag {4}
$$
Figure 3: The framework of the Subspace Identification Guarantee model. The pre-trained backbone networks are used to extract the feature  $f$  from observed data. The bottleneck and  $f_{\mu}, f_{\sigma}$  are used to generate  $\mathbf{z}$  with a reparameterization trick. Label predictor takes  $\mathbf{z}_2, \mathbf{z}_3$ , and  $\mathbf{u}$  as input to model  $p_{\mathbf{y}|\mathbf{u}, \mathbf{z}_2, \mathbf{z}_3}$ . The decoder is used to model the marginal distribution. Finally,  $\mathbf{z}_2$  is used for class-aware conditional alignment.
$h_{1,i}$  such that  $z_{1,i} = h_{1,i}(\hat{\mathbf{z}}_1,\hat{\mathbf{z}}_2)$ . And for each  $z_{3,i},i\in \{n_1 + n_2 + 1,\dots ,n_1 + n_2 + n_3\}$ , there exists a  $h_{3,i}$  such that  $z_{3,i} = h_{3,i}(\hat{\mathbf{z}}_2,\hat{\mathbf{z}}_3)$ . Then we prove that  $\mathbf{z}_4$  is block-wise identifiable, which means that there exists an invertible function  $h_4$ , s.t.  $\mathbf{z}_4 = h_4(\hat{\mathbf{z}}_4)$ .
Lemma 2. [30] Following the data generation process in Section 2.1 and the assumptions A4-A6 in Theorem 3, we further make the following assumption:
- A7 (Domain Variability: For any set  $A_{\mathbf{z}} \subseteq \mathcal{Z}$ ) with the following two properties: 1)  $A_{\mathbf{z}}$  has nonzero probability measure, i.e.  $\mathbb{P}[\mathbf{z} \in A_{\mathbf{z}} | \{\mathbf{u} = \mathbf{u}', \mathbf{y} = \mathbf{y}'\}] > 0$  for any  $\mathbf{u}' \in \mathcal{U}$  and  $\mathbf{y}' \in \mathcal{V}$ . 2)  $A_{\mathbf{z}}$  cannot be expressed as  $B_{\mathbf{z}_4} \times \mathcal{Z}_1 \times \mathcal{Z}_2 \times \mathcal{Z}_3$  for any  $B_{\mathbf{z}_4} \subset \mathcal{Z}_4$ .
$\exists \mathbf{u}_1, \mathbf{u}_2 \in \mathcal{U}$  and  $\mathbf{y}_1, \mathbf{y}_2 \in \mathcal{V}$ , such that  $\int_{\mathbf{z} \in \mathcal{A}_{\mathbf{z}}} p_{\mathbf{z}} |_{\mathbf{u}_1, \mathbf{y}_1} d\mathbf{z} \neq \int_{\mathbf{z} \in \mathcal{A}_{\mathbf{z}}} p_{\mathbf{z}} |_{\mathbf{u}_2, \mathbf{y}_2} d\mathbf{z}$ . By modeling the data generation process in Section 2.1, the  $\mathbf{z}_4$  is block-wise identifiable.
The proof can be found in Appendix B.3. Lemma 4 shows that  $\mathbf{z}_4$  can be block-wise identifiable when the  $p_{\mathbf{z}|\mathbf{u}}$  changes sufficiently across domains.
In summary, we can obtain the estimated latent variables  $\hat{\mathbf{z}}$  with the help of subspace identification and block-wise identification.
4 Subspace Identification Guarantee Model
Based on the theoretical results, we proposed the Subspace Identification Guaranteed model (SIG) as shown in Figure 3, which contains a variational-inference-based neural architecture to model the marginal distribution and a class-aware conditional alignment to mitigate the target shift.
4.1 Variational-Inference-based Neural Architecture