SlowGuess/NeurIPS_2023_OCR huggingface dataset:
According to the data generation process in Figure 1, we first derive the evidence lower bound (ELBO) in Equation (5).
$$
\begin{array}{l} E L B O = \mathbb {E} _ {q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x})} \ln p _ {\mathbf {x} | \mathbf {z}} (\mathbf {x} | \mathbf {z}) + \mathbb {E} _ {q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x})} \ln p _ {\mathbf {y} | \mathbf {u}, \mathbf {z} _ {2}, \mathbf {z} _ {3}} (\mathbf {y} | \mathbf {u}, \mathbf {z} _ {2}, \mathbf {z} _ {3}) \\ + \mathbb {E} _ {q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x})} \ln p _ {\mathbf {u} | \mathbf {z}} (\mathbf {u} | \mathbf {z}) - D _ {K L} \left(q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x}) \| p _ {\mathbf {z}} (\mathbf {z})\right). \tag {5} \\ \end{array}
$$
Since the reconstruction of  $\mathbf{u}$  is not the optimization goal, we remove the reconstruction of  $\mathbf{u}$  and we rewrite Equation (5) as the objective function in Equation (6).
$$
\mathcal {L} _ {e l b o} = \mathcal {L} _ {v a e} + \mathcal {L} _ {y}
$$
$$
\mathcal {L} _ {v a e} = - \mathbb {E} _ {q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x})} \ln p _ {\mathbf {x} | \mathbf {z}} (\mathbf {x} | \mathbf {z}) + D _ {K L} \left(q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x}) \| p _ {\mathbf {z}} (\mathbf {z})\right) \tag {6}
$$
$$
\mathcal {L} _ {y} = - \mathbb {E} _ {q _ {\mathbf {z} | \mathbf {x}} (\mathbf {z} | \mathbf {x})} \ln p _ {\mathbf {y} | \mathbf {u}, \mathbf {z} _ {2}, \mathbf {z} _ {3}} (\mathbf {y} | \mathbf {u}, \mathbf {z} _ {2}, \mathbf {z} _ {3}).
$$
To minimize the pairwise class confusion, we further employ the minimum class confusion [25] into the classification loss  $\mathcal{L}_y$ . According to the objective function in Equation (6), we illustrate how to
implement the SIG model in Figure 3. First, we take the observed data  $\mathbf{x}_{S_i}$  and  $\mathbf{x}_T$  from the source domains and the target domain as the inputs of the pre-trained backbone networks like ResNet50 and extract the feature  $f_{S_i}$  and  $f_{\mathcal{T}}$ . Sequentially, we employ an MLP-based encoder, which contains bottleneck networks,  $f_{\mu}$  and  $f_{\sigma}$ , to extract the latent variables  $\mathbf{z}$ . Then, we take  $\mathbf{z}$  to reconstruct the pre-trained features via an MLP-based decoder to estimate the marginal distribution  $p_{\mathbf{x}|\mathbf{u}}$ . Finally, we take the  $\mathbf{z}_2, \mathbf{z}_3$ , and the domain embedding to predict the source label to estimate  $p_{\mathbf{y}|\mathbf{u},\mathbf{z}_2,\mathbf{z}_3}$ .
4.2 Class-aware Conditional Alignment
To estimate the target label distribution  $p_{\mathbf{y}|\mathbf{u}\tau}$  and mitigate the influence of target shift, we propose the class-aware conditional alignment to automatically adjust the conditional alignment loss for each sample. Formally, it can be written as
$$
\mathcal {L} _ {a} = \frac {1}{C} \sum_ {i = 1} ^ {C} w ^ {(i)} \cdot p _ {\hat {\mathbf {y}} ^ {(i)}} | | \hat {\mathbf {z}} _ {3, S} ^ {(i)} - \hat {\mathbf {z}} _ {3, T} ^ {(i)} | | _ {2}, \quad w ^ {(i)} = 1 + \exp (- H (p _ {\hat {\mathbf {y}} ^ {(i)}})), \tag {7}
$$
where  $C$  denotes the class number;  $\hat{\mathbf{z}}_{3,\mathcal{S}}^{(i)}$  and  $\hat{\mathbf{z}}_{3,\mathcal{T}}^{(i)}$  denote the latent variables of ith class from source and target domain, respectively;  $w^{(i)}$  denotes the prediction uncertainty of each class in the target domain;  $p_{\hat{\mathbf{Y}}^{(i)}}$  denotes the estimated label probability density of ith class;  $H$  denotes the entropy.
The aforementioned class-aware conditional alignment is based on the existing conditional alignment loss, which can be formalized in Equation (8).
$$
\mathcal {L} _ {a} = \frac {1}{C} \sum_ {i = 1} ^ {C} \left| \left| \hat {\mathbf {z}} _ {3, S} ^ {(i)} - \hat {\mathbf {z}} _ {3, T} ^ {(i)} \right| \right| _ {2}, \tag {8}
$$
However, conventional conditional alignment usually suffers from two drawbacks including misestimated centroid and low-quality pseudo-labels. First, the conditional alignment method implicitly assumes that the feature centroids from different domains are the same. But it is hard to estimate the correct centroids of the target domain for the class with low probability density. Second, conditional alignment heavily relies on the quality of the pseudo label. But existing methods usually use pseudo labels without any discrimination, which might result in false alignment. To solve these problems, we consider two types of reweighting.
Distribution-based Reweighting for Misestimated Centroid: Although the conditional alignment method implicitly assumes that the feature centroids from different domains are the same, it is hard to estimate the correct centroids of the target domain for the class with low probability density. To address this challenge, one straightforward solution is to consider the label distribution of the target domain. To achieve this, we employ the technique of black box shift estimation method (BBSE) [37] to estimate the label distribution from the target domain  $p_{\hat{\mathbf{y}}}$ . So we use the estimated label distribution to reweight the conditional alignment loss in Equation (8).
Entropy-based Reweighting for Low-quality Pseudo-labels: conditional alignment heavily relies on the quality of the pseudo label. However, existing methods usually use pseudo labels without any discrimination, which might result in false alignment. To address this challenge, we consider the prediction uncertainty of each class in the target domain. Technologically, for each sample in the target dataset, we calculate the entropy-based weights via the prediction results which are shown as  $w^{(i)}$  in Equation (7).
By combining the distribution-based weights and the entropy-based weight, we can obtain the class-aware conditional alignment as shown in Equation (7). Hence the total loss of the Subspace Identification Guarantee model can be formalized as follows:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {y} + \beta \mathcal {L} _ {v a e} + \alpha \mathcal {L} _ {a}, \tag {9}
$$
where  $\alpha, \beta$  denote the hyper-parameters.
5.1 Experiments on Simulation Data
In this subsection, we illustrate the experiment results of simulation data to evaluate the theoretical results of subspace identification in practice.
Data Generation. We generate the simulation data for binary classification with 8 domains. To better evaluate our theoretical results, we follow the data generation process in Figure 2, which includes two types of latent variables, i.e., domain-specific latent variables  $\mathbf{z}_s$  and domain-invariant latent variables  $\mathbf{z}_c$ . We let the dimensions of  $\mathbf{z}_s$  and  $\mathbf{z}_c$  be both 2. Moreover,  $\mathbf{z}_s$  are sampled from  $u$  different mixture of Gaussians, and  $\mathbf{z}_c$  are sampled from a factorized Gaussian distribution. We let the data generation process from latent variables to observed variables be MLPs with the Tanh activation function. We further split the simulation dataset into the training set, validation set, and test set.
Evaluation Metrics. First, we employ the accuracy of the target domain data to measure the classification performance of the model. Second, we compute the Mean Correlation Coefficient (MCC) between the ground-truth  $z_{s}$  and the estimated  $\hat{z}_{s}$  on the test dataset to evaluate the component-wise identifiability of domain-specific latent variables. A higher MCC denotes the better identification performance the model can achieve. Third, to evaluate the performance of subspace identifiability of domain-specific latent variables, we first use the estimated  $\hat{z}_{s}$  from the validation set to regress each dimension of the ground-truth  $z_{s}$  from the validation set with the help of a MLPs. Sequentially, we take the  $\hat{z}_{s}$  from the test set as input to estimate how well the MLPs can reconstruct the ground-truth  $z_{s}$  from the test set, so we employ Root Mean Square Error (RMSE) to measure the extent of subspace identification. A low RMSE denotes that there exists a transformation  $h_{i}$  between  $z_{s,i}$  and  $\hat{z}_{s,1}, \hat{z}_{s,2}$ , i.e.  $z_{s,i} = h_{i}(\hat{z}_{s,1},\hat{z}_{s,2})$ ,  $i \in \{1,2\}$ . For the scenario where the number of domains is less than 8, we first fix the target domain and then try all the combinations of the source domains. And we publish the average performance of all the combinations. We repeat each experiment over 3 random seeds.
5.1.2 Results and Discussion
The experimental results of the simulation dataset are shown in Table 1. According to the experiment results, we can obtain the following conclusions: 1) We can find
Table 1: Experiments results on simulation data.
State	U	ACC	MCC	RMSE
Component-wise Identification	8	0.9982(0.0004)	0.9037(0.0087)	0.0433(0.0051)
6	0.9982(0.0007)	0.8976(0.0162)	0.0439(0.0073)
5	0.9982(0.0007)	0.8973(0.0131)	0.0441(0.0055)
Subspace Identification	4	0.9233(0.2039)	0.8484(0.1452)	0.0582(0.0431)
3	0.8679(0.2610)	0.8077(0.1709)	0.0669(0.0482)
No Identification	2	0.5978(0.3039)	0.6184(0.2093)	0.1272(0.0608)
that the values of MCC increase along with the number of domains. Moreover, the values of MCC are high (around 0.9) and stable when the number of domains is larger than 5. This result corresponds to the theoretical result of component-wise identification, where a certain number of domains (i.e.  $2n + 1$ ) are necessary for component-wise identification. 2) We can find that the values of RMSE decrease along with the number of domains. Furthermore, the values of RMSE are low and stable (less than 0.07) when the number of domains is larger than 3, but it drops sharply when  $u = 2$ . These experimental results coincide with the theoretical results of subspace identification as well as the intuition where a certain number of domains are necessary for subspace identification (i.e.  $n_s + 1$ ). 3) According to the experimental results of ACC, we can find that the accuracy grows along with the number of domains and its changing pattern is relevant to that of RMSE, i.e., the performance is stable when the number of domains is larger than 3. The ACC results also indirectly support the results of subspace identification, since one straightforward understanding of subspace identification is that the domain-specific information is preserved in  $\hat{z}_s$ . And the latent variables are well disentangled with the help of subspace identification, which benefits the model performance.
5.2 Experiments on Real-world Data
Datasets: We consider four benchmarks: Office-Home, PACS, ImageCLEF, and DomainNet. For each dataset, we let each domain be a target domain and the other domains be the source domains. For the DomainNet dataset, we equip a cross-attention module to the ResNet101 backbone networks for
Table 2: Classification results on the Office-Home and ImageCLEF datasets. For the Office-Home dataset, We employ ResNet50 as the backbone network. For the ImageCLEF dataset, we employ ResNet18 as the backbone network.
Model	Office-Home	ImageCLEF
Art	Clipart	Product	RealWorld	Average	P	C	I	Average
Source Only [16]	64.5	52.3	77.6	80.7	68.8	77.2	92.3	88.1	85.8
DANN [11]	64.2	58.0	76.4	78.8	69.3	77.9	93.7	91.8	87.8
DAN [40]	68.2	57.9	78.4	81.9	71.6	77.6	93.3	92.2	87.7
DCTN [69]	66.9	61.8	79.2	77.7	71.4	75.0	95.7	90.3	87.0
MFSAN [81]	72.1	62.0	80.3	81.8	74.1	79.1	95.4	93.6	89.4
WADN [54]	75.2	61.0	83.5	84.4	76.1	77.7	95.8	93.2	88.9
iMSDA [30]	75.4	61.4	83.5	84.4	76.2	79.2	96.3	94.3	90.0
SIG	76.4	63.9	85.4	85.8	77.8	79.3	97.3	94.3	90.3
better usage of domain knowledge. We also employ the alignment of MDD [78]. For the Office-Home and ImageCLEF datasets, we employ the pre-trained ResNet50 with an MLP-based classifier. For the PACS dataset, we use ResNet18 with an MLP-based classifier. The implementation details are provided in the Appendix C. We report the average results over 3 random seeds.
Baselines: Besides the classical approaches for single source domain adaptation like DANN [11], DAN [40], MCD [50], and ADDA [61]. We also compare our method with several state-of-the-art multi-source domain adaptation methods, for example, MIAN-  $\gamma$  [45], T-SVDNet [33], LtC-MSDA [63], SPS [64], and PFDA [10]. Moreover, we further consider the WADN [54], which is devised for the target shift of multi-source domain adaptation. For a fair comparison, we employ the same pre-train backbone networks instead of the pre-trained features for WADN in the original paper. We also consider the latest
Table 3: Classification results on the PACS datasets. We employ ResNet18 as the backbone network. Experiment results of other compared methods are taken from ([30]).
Model	A	C	P	S	Average
Source Only [16]	74.9	72.1	94.5	64.7	76.7
DANN [11]	81.9	77.5	91.8	74.6	81.5
MDAN [79]	79.1	76.0	91.4	72.0	79.6
WBN [43]	89.9	89.7	97.4	58.0	83.8
MCD [50]	88.7	88.9	96.4	73.9	87.0
M3SDA [46]	89.3	89.9	97.3	76.7	88.3
CMSS [70]	88.6	90.4	96.9	82.0	89.5
LtC-MSDA [63]	90.1	90.4	97.2	81.5	89.8
T-SVDNet [33]	90.4	90.6	98.5	85.4	91.2
iMSDA [30]	93.7	92.4	98.4	89.2	93.4
SIG	94.1	93.6	98.6	89.5	93.9
iMSDA [30], which addresses the MSDA via component-wise identification.
5.2.2 Results and Discussion
Experimental results on Office-Home, ImageCLEF, PACS, and DomainNet are shown in Table 2, 3, and 4, respectively. Experiment results of other compared methods are provided in Appendix D.2.
According to the experiment results of the Office-Home dataset on the left side of Table 2, our SIG model significantly outperforms all other baselines on all the transfer tasks. Specifically, our method outperforms the most competitive baseline by a clear margin of  $1.3\% - 4\%$  and promotes the classification accuracy substantially on the hard transfer task, e.g. Clipart. It is noted that our method achieves a better performance than that of WADN, which is designed for the target shift scenario. This is because our method not only considers how the domain variables influence the distribution of labels but also identifies the latent variables of the data generation process. Moreover, our SIG method also outperforms iMSDA, indirectly reflecting that the proposed data generation process is closer to the real-world setting and the subspace identification can achieve better disentanglement performance under limited auxiliary variables.
For datasets like ImageCLEF and PACS, our method also achieves the best-averaged results. In detail, we achieved a comparable performance in all the transfer tasks in the ImageCLFE dataset. In the PACS dataset, our SIG method still performs better than the latest compared methods like iMSDA and T-SVDNet in some challenging transfer tasks like Cartoon. Finally, we also consider the
Table 4: Classification results on the DomainNet datasets. We employ ResNet101 as the backbone network. Experiment results of other compared methods are taken from ([34] and [64]).
Model	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Average
Source Only [16]	52.1	23.4	47.6	13.0	60.7	46.5	40.6
ADDA [61]	47.5	11.4	36.7	14.7	49.1	33.5	32.2
MCD [50]	54.3	22.1	45.7	7.6	58.4	43.5	38.5
DANN [11]	60.6	25.8	50.4	7.7	62.0	51.7	43
DCTN [69]	48.6	23.5	48.8	7.2	53.5	47.3	38.2
\( M^3SDA-β \) [46]	58.6	26.0	52.3	6.3	62.7	49.5	42.6
ML_MSDA [35]	61.4	26.2	51.9	19.1	57.0	50.3	44.3
meta-MCD [32]	62.8	21.4	50.5	15.5	64.6	50.4	44.2
LtC-MSDA [63]	63.1	28.7	56.1	16.3	66.1	53.8	47.4
CMSS [70]	64.2	28.0	53.6	16.9	63.4	53.8	46.5
DRT+ST [34]	71.0	31.6	61.0	12.3	71.4	60.7	51.3
SPS [64]	70.8	24.6	55.2	19.4	67.5	57.6	49.2
PFDA [10]	64.5	29.2	57.6	17.2	67.2	55.1	48.5
iMSDA [30]	68.1	25.9	57.4	17.3	64.2	52.0	47.5
SIG	72.7	32.0	61.5	20.5	72.4	59.5	53.0
Figure 4: Ablation study on the Office-Home dataset. we explore the impact of different loss terms.
most challenging dataset, DomainNet, which contains more classes and more complex domain shifts. Results in Table 4 show the significant performance of the proposed SIG method, which provides  $3.3\%$  averaged promotion, although the performance in the task of Sketch is slightly lower than that of DRT+ST. Compared with iMSDA, our SIG overpasses by a large margin under a more general data generation process.
Ablation Study: To evaluate the effectiveness of individual loss terms, we also devise the two model variants. 1) SIG-sem: remove the class-aware alignment loss. 2) SIG-vae: remove the reconstruction loss and the KL divergence loss. Experiment results on the Office-Home dataset are shown in Figure 4. We can find that the class-aware alignment loss plays an important role in the model performance, reflecting that the class-aware alignment can mitigate the influence of target shift. We also discover that incorporating the reconstruction and KL divergence has a positive impact on the overall performance of the model, which shows the necessity of modeling the marginal distributions.
This paper presents a general data generation process for multi-source domain adaptation, which coincides with real-world scenarios. Based on this data generation process, we prove that the changing latent variables are subspace identifiable, which provides a novel solution for disentangled representation. Compared with the existing methods, the proposed subspace identification theory requires fewer auxiliary variables and frees the model from the monotonic transformation of latent variables, making it possible to apply the proposed method to real-world data. Experiment results on several mainstream benchmark datasets further evaluate the effectiveness of the proposed subspace identification guaranteed model. In summary, this paper takes a meaningful step for causal representation learning. Broader Impacts: SIG disentangles the latent variables to create a model that is more transparent, thereby aiding in the reduction of bias and the promotion of fairness. Limitation: However, the proposed subspace identification still requires several assumptions that might not be met in real-world scenarios. Therefore, how further to relax the assumptions, i.e., conditional independent assumption, would be an interesting future direction.
We are very grateful to the anonymous reviewers for their help in improving the paper. This research was supported in part by the National Key R&D Program of China (2021ZD0111501), the National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052), the major key project of PCL (PCL2021A12). This project is also partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
[1] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734, 2019.
[2] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems, 29, 2016.
[3] Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In IJCAI: proceedings of the conference, volume 2019, page 2060. NIH Public Access, 2019.
[4] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 7354-7362, 2019.
[5] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3296-3303, 2019.
[6] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 627-636, 2019.
[7] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pages 1081-1090. PMLR, 2019.
[8] Yuansi Chen and Peter Buhlmann. Domain adaptation under structural causal models. The Journal of Machine Learning Research, 22(1):11856-11935, 2021.
[9] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
[10] Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16654-16663, 2021.
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[12] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
[13] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pages 2839-2848. PMLR, 2016.
[82] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE transactions on neural networks and learning systems, 32(4):1713-1722, 2020.