SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nils Sturma
Technical University of Munich Munich Center for Machine Learningnils.sturma@tum.de
Mathias Drton
Technical University of Munich Munich Center for Machine Learning mathias.drton@tum.de
LIDS, Massachusetts Institute of Technology  
Broad Institute of MIT and Harvard  
csquires@mit.edu
LIDS, Massachusetts Institute of Technology  
Broad Institute of MIT and Harvard  
cuhler@mit.edu
The goal of causal representation learning is to find a representation of data that consists of causally related latent variables. We consider a setup where one has access to data from multiple domains that potentially share a causal representation. Crucially, observations in different domains are assumed to be unpaired, that is, we only observe the marginal distribution in each domain but not their joint distribution. In this paper, we give sufficient conditions for identifiability of the joint distribution and the shared causal graph in a linear setup. Identifiability holds if we can uniquely recover the joint distribution and the shared causal representation from the marginal distributions in each domain. We transform our results into a practical method to recover the shared latent causal graph.
An important challenge in machine learning is the integration and translation of data across multiple domains (Zhu et al., 2017; Zhuang et al., 2021). Researchers often have access to large amounts of unpaired data from several domains, e.g., images and text. It is then desirable to learn a probabilistic coupling between the observed marginal distributions that captures the relationship between the domains. One approach to tackle this problem is to assume that there is a latent representation that is invariant across the different domains (Bengio et al., 2013; Ericsson et al., 2022). Finding a probabilistic coupling then boils down to learning such a latent representation, that is, learning high-level, latent variables that explain the variation of the data within each domain as well as similarities across domains.
In traditional representation learning, the latent variables are assumed to be statistically independent, see for example the literature on independent component analysis (Hyvärinen and Oja, 2000; Comon and Jutten, 2010; Khemakhem et al., 2020). However, the assumption of independence can be too stringent and a poor match to reality. For example, the presence of clouds and the presence of wet roads in an image may be dependent, since clouds may cause rain which may in turn cause wet roads. Thus, it is natural to seek a causal representation, that is, a set of high-level causal variables and relations among them (Schölkopf et al., 2021; Yang et al., 2021b). Figure 1 illustrates the setup of multi-domain causal representation learning, where multiple domains provide different views on a shared causal representation.
Our motivation to study multi-domain causal representations comes, in particular, from single-cell data in biology. Given a population of cells, different technologies such as imaging and sequencing
provide different views on the population. Crucially, since these technologies destroy the cells, the observations are uncoupled, i.e., a specific cell may either be used for imaging or sequencing but not both. The aim is to integrate the different views on the population to study the underlying causal mechanisms determining the observed features in various domains (Butler et al., 2018; Stuart et al., 2019; Liu et al., 2019; Yang et al., 2021a; Lopez et al., 2022; Gossi et al., 2023; Cao et al., 2022). Unpaired multi-domain data also appears in many applications other than single-cell biology. For example, images of similar objects are captured in different environments (Beery et al., 2018), large biomedical and neuroimaging data sets are collected in different domains (Miller et al., 2016; Essen et al., 2013; Shafto et al., 2014; Wang et al., 2003), or stocks are traded in different markets.
In this paper, we study identifiability of the shared causal representation, that is, its uniqueness in the infinite data limit. Taking on the same perspective as, for example, in Schölkopf et al. (2021) and Squires et al. (2023), we assume that observed data is generated in two steps. First, the latent variables  $Z = (Z_{i})_{i\in \mathcal{H}}$  are sampled from a distribution  $P_Z$ , where  $P_Z$  is determined by an unknown structural causal model among the latent variables. Then, in each domain  $e\in \{1,\dots,m\}$ , the observed vector  $X^{e}\in \mathbb{R}^{d_{e}}$  is the image of a subset of the latent variables under a domain-specific, injective mixing function  $g_{e}$ . That is,
$$
X ^ {e} = g _ {e} \left(Z _ {S _ {e}}\right),
$$
where  $S_{e} \subseteq \mathcal{H}$  is a subset of indices. A priori, it is unknown whether a latent variable  $Z_{i}$  with  $i \in S_{e}$  is shared across domains or domain-specific. Even the number of latent variables which are shared across domains is unknown. Moreover, we only observe the marginal distribution of each random vector  $X^{e}$ , but none of the joint distributions over pairs  $X^{e}, X^{f}$  for  $e \neq f$ . Said differently, observations across domains are unpaired. Assuming that the structural causal model among the latent variables as well as the mixing functions are linear, our main contributions are:
1. We lay out conditions under which we can identify the joint distribution of  $X^1, \ldots, X^m$ .
2. We give additional conditions under which we are able to identify the causal structure among the shared latent variables.
In particular, identifiability of the joint distribution across domains enables data translation. That is, given observation  $x$  in domain  $e$ , translation to domain  $f$  can be achieved by computing  $\mathbb{E}[X_f|X_e = x]$ . Furthermore, identifying the causal structure among the shared latent variables lets us study the effect of interventions on the different domains.
The main challenge in proving rigorous identifiability results for multi-domain data is that we cannot apply existing results for single-domain data in each domain separately. Even if the causal structure of the latent variables in a single domain is identifiable, it remains unclear how to combine multiple causal structures, i.e., in which way latent variables are shared. We circumvent this problem via a two-step approach: First, we extend the identifiability of linear independent component analysis (Comon, 1994; Hyvarinen and Oja, 2000; Eriksson and Koivunen, 2004; Mesters and Zwiernik, 2022) to the multi-domain setup, which allows us to identify the joint distribution and distinguish between shared and domain-specific latent variables. Moreover, we identify an "overall mixing matrix" and, in a second step, exploit sparsity constraints in this matrix to identify the causal structure among the shared latent variables. This leverages recent results on causal discovery under measurement error in single domains that also exploit sparsity (Xie et al., 2020; Chen et al., 2022; Xie et al., 2022; Huang et al., 2022). Although we emphasize that our focus in this paper is on identifiability, our proofs also suggest methods to learn the joint distribution as well as the shared causal graph from finite samples. We provide algorithms for the noisy setting and, moreover, we analyze how the number of domains reduce uncertainty with respect to the learned representation.
The paper is organized as follows. In the next paragraphs we discuss further related work. Section 2 provides a precise definition of the considered setup. In Section 3 we consider identifiability of the joint distribution. Using these results, we study identifiability of the causal graph in Section 4. We conclude with a small simulation study as a proof of concept for the finite sample setting in Section 5. Due to space constraints, the detailed discussion of the finite sample setting is deferred to the Appendix. Moreover, the Appendix contains all proofs, discussions on the necessity of our assumptions, and additional examples and simulation results.
Multi-domain Integration. Motivated by technological developments for measuring different modalities at single-cell resolution, several methods have been proposed recently for domain
Figure 1: Setup. A latent causal representation where multiple domains  $X^e$  provide different "views" on subsets of the latent variables  $Z_i$ . The domains may correspond to different data modalities such as images, text or numerical data. Crucially, the observations across domains are unpaired, i.e., they arise from different states of the latent causal model.
translation between unpaired data. The proposed methods rely on a variety of techniques, including manifold alignment (Welch et al., 2017; Amodio and Krishnaswamy, 2018; Liu et al., 2019), matrix factorization (Duren et al., 2018), correlation analysis (Barkas et al., 2019; Stuart et al., 2019), coupled autoencoders (Yang and Uhler, 2019), optimal transport (Cao et al., 2022), regression analysis (Yuan and Duren, 2022), and semisupervised learning (Lin et al., 2022). Implicitly, these methods presume the existence of a shared latent space where the different modalities either completely align or at least overlap. However, to the best of our knowledge, none of these methods have rigorous identifiability guarantees, i.e., the methods are not guaranteed to recover a correct domain translation mapping even for infinite data. Our work advances the theoretical understanding of multi-domain integration by providing identifiability guarantees on recovering the shared latent space.
Group Independent Component Analysis. The primary tool that we use for identifiability is linear independent component analysis (ICA) (Comon, 1994; Eriksson and Koivunen, 2004). Many works extend ICA to the multi-domain setting. These methods primarily come from computational neuroscience, where different domains correspond to different subjects or studies. However, to the best of our knowledge, all prior works require pairing between samples. These works can be categorized based on whether the samples are assumed to be voxels (Calhoun et al., 2001; Esposito et al., 2005), time points (Svensén et al., 2002; Varoquaux et al., 2009; Hyvarinen and Ramkumar, 2013), or either (Beckmann and Smith, 2005; Sui et al., 2009). For reviews, see Calhoun et al. (2009) and Chabriel et al. (2014). Related are methods for independent vector analysis (Kim et al., 2006; Anderson et al., 2014; Bhinge et al., 2019) and multiset canonical correlation analysis (Nielsen, 2002; Li et al., 2011; Klami et al., 2014), which allow the latent variables to take on different values in each domain but still require sample pairing. Most of the mentioned methods lack identifiability guarantees, only newer work (Richard et al., 2021) provides sufficient conditions for identifiability. Furthermore, all mentioned methods assume that every latent variable is shared across all domains, while our setup allows for shared and domain-specific latent variables. Some methods, e.g., Lukic et al. (2002), Maneshi et al. (2016), and Pandeva and Forre (2023), permit both shared and domain-specific components, but only consider the paired setting. In this paper, we extend these results to the unpaired setting.
Latent Structure Discovery. Learning causal structure between latent variables has a long history, e.g., in measurement models (Silva et al., 2006). One recent line of work studies the problem under the assumption of access to interventional data (e.g., Liu et al., 2022; Squires et al., 2023). In particular, Squires et al. (2023) show that the latent graph is identifiable if the interventions are sufficiently diverse. Another line of work, closer to ours and not based on interventional data, shows that the graph is identified under certain sparsity assumptions on the mixing functions (Xie et al., 2020; Chen et al., 2022; Xie et al., 2022; Huang et al., 2022). However, these methods are not suitable in our setup since they require paired data in a single domain. One cannot apply them in each domain separately since it would be unclear how to combine the multiple latent causal graphs, that is, which of the latent variables are shared. In this work, we lay out sparsity assumptions on the mixing
functions that are tailored to the unpaired multi-domain setup. The works of Adams et al. (2021) and Zeng et al. (2021) may be considered closest to ours as they also treat a setting with multiple domains and unpaired data. However, our setup and results are more general. Adams et al. (2021) assume that the number of observed variables are the same in each domain, whereas we consider domains of different dimensions corresponding to the fact that observations may be of very different nature. Further, we allow for shared and domain-specific latent variables, where the number of shared latent variables is unknown, while in Adams et al. (2021) it is assumed that all latent variables are shared. Compared to Zeng et al. (2021), we consider a general but fixed number of observed variables, while Zeng et al. (2021) only show identifiability of the full model in a setup where the number of observed variables in each domain increases to infinity. On a more technical level, the conditions in Zeng et al. (2021) require two pure children to identify the shared latent graph, while we prove identifiability under the weaker assumption of two partial pure children; see Section 4 for precise definitions.
Notation. Let  $\mathbb{N}$  be the set of nonnegative integers. For positive  $n\in \mathbb{N}$ , we define  $[n] = \{1,\dots ,n\}$ . For a matrix  $M\in \mathbb{R}^{a\times b}$ , we denote by  $M_{A,B}$  the submatrix containing the rows indexed by  $A\subseteq [a]$  and the columns indexed by  $B\subseteq [b]$ . Moreover, we write  $M_B$  for the submatrix containing all rows but only the subset of columns indexed by  $B$ . Similarly, for a tuple  $x = (x_{1},\ldots ,x_{b})$ , we denote by  $x_{B}$  the tuple only containing the entries indexed by  $B$ . A matrix  $Q = Q_{\sigma}\in \mathbb{R}^{p\times p}$  is a signed permutation matrix if it can be written as the product of a diagonal matrix  $D$  with entries  $D_{ii}\in \{\pm 1\}$  and a permutation matrix  $\widetilde{Q}_{\sigma}$  with entries  $(\widetilde{Q}_{\sigma})_{ij} = \mathbb{1}_{j = \sigma (i)}$ , where  $\sigma$  is a permutation on  $p$  elements. Let  $P$  be a  $p$ -dimensional joint probability measure of a collection of random variables  $Y_{1},\ldots ,Y_{p}$ . Then we denote by  $P_{i}$  the marginal probability measure such that  $Y_{i}\sim P_{i}$ . We say that  $P$  has independent marginals if the random variables  $Y_{i}$  are mutually independent. Moreover, we denote by  $M\# P$  the  $d$ -dimensional push-forward measure under the linear map defined by the matrix  $M\in \mathbb{R}^{d\times p}$ . If  $Q$  is a signed permutation matrix and the probability measure  $P$  has independent marginals, then  $Q\# P$  also has independent marginals. For univariate probability measures we use the shorthand  $(-1)\# P = -P$ .
Let  $\mathcal{H} = [h]$  for  $h\geq 1$ , and let  $Z = (Z_{1},\ldots ,Z_{h})$  be latent random variables that follow a linear structural equation model. That is, the variables are related by a linear equation
$$
Z = A Z + \varepsilon , \tag {1}
$$
with  $h \times h$  parameter matrix  $A = (a_{ij})$  and zero-mean, independent random variables  $\varepsilon = (\varepsilon_1, \ldots, \varepsilon_h)$  that represent stochastic errors. Assume that we have observed random vectors  $X^e \in \mathbb{R}^{d_e}$  in multiple domains of interest  $e \in [m]$ , where the dimension  $d_e$  may vary across domains. Each random vector is the image under a linear function of a subset of the latent variables. In particular, we assume that there is a subset  $\mathcal{L} \subseteq \mathcal{H}$  representing the shared latent space such that each  $X^e$  is generated via the mechanism
$$
X ^ {e} = G ^ {e} \cdot \left( \begin{array}{c} Z _ {\mathcal {L}} \\ Z _ {I _ {e}} \end{array} \right), \tag {2}
$$
where  $I_{e} \subseteq \mathcal{H} \setminus \mathcal{L}$ . We say that the latent variable  $Z_{I_e}$  are domain-specific for domain  $e \in [m]$  while the latent variables  $Z_{\mathcal{L}}$  are shared across all domains. As already noted, we are motivated by settings where the shared latent variables  $Z_{\mathcal{L}}$  capture the key causal relations and the different domains are able to give us combined information about these relations. Likewise, we may think about the domain-specific latent variables  $Z_{I_e}$  as "noise" in each domain, independent of the shared latent variables. Specific models are now derived from (1)-(2) by assuming specific (but unknown) sparsity patterns in  $A$  and  $G^{e}$ . Each model is given by a "large" directed acyclic graph (DAG) that encodes the multi-domain setup. To formalize this, we introduce pairwise disjoint index sets  $V_{1},\ldots ,V_{m}$ , where  $V_{e}$  indexes the coordinates of  $X^{e}$ , i.e.,  $X^{e} = (X_{v}:v\in V_{e})$  and  $|V_{e}| = d_{e}$ . Then  $V = V_{1}\cup \dots \cup V_{m}$  indexes all observed random variables. We define an  $m$ -domain graph such that the latent nodes are the only parents of observed nodes and there are no edges between shared and domain-specific latent nodes.
Definition 2.1. Let  $\mathcal{G}$  be a DAG whose node set is the disjoint union  $\mathcal{H} \cup V = \mathcal{H} \cup V_1 \cup \dots \cup V_m$ . Let  $D$  be the edge set of  $\mathcal{G}$ . Then  $\mathcal{G}$  is an  $m$ -domain graph with shared latent nodes  $\mathcal{L} = [\ell] \subseteq \mathcal{H}$  if the following is satisfied:
1. All parent sets contain only latent variables, i.e.,  $\mathrm{pa}(v) = \{w : w \to v \in D\} \subseteq \mathcal{H}$  for all  $v \in \mathcal{H} \cup V$ .
Figure 2: Compact version of a 2-domain graph  $\mathcal{G}_2$  with five latent nodes and two domains  $V_{1}$  and  $V_{2}$ . All observed nodes in each domain are represented by a single grey node. We draw a dashed blue edge from latent node  $h\in \mathcal{H}$  to domain  $V_{e}\subseteq V$  if  $h\in S_e = \mathrm{pa}(V_e)$ . The random vectors associated to the two domains are uncoupled, that is, we do not observe their joint distribution.
2. The set  $\mathcal{L}$  consists of the common parents of variables in all different domains, i.e.,  $u\in \mathcal{L}$  if and only if  $u\in \mathrm{pa}(v)\cap \mathrm{pa}(w)$  for  $v\in V_e,w\in V_f$  with  $e\neq f$
3. Let  $I_{e} = S_{e} \setminus \mathcal{L}$  be the domain-specific latent nodes, where  $S_{e} := \mathrm{pa}(V_{e}) = \cup_{v \in V_{e}} \mathrm{pa}(v) \subseteq \mathcal{H}$ . Then there are no edges in  $D$  that connect a node in  $\mathcal{L}$  and a node  $\cup_{e=1}^{m} I_{e}$  or that connect a node in  $I_{e}$  and a node in  $I_{f}$  for any  $e \neq f$ .
To emphasize that a given DAG is an  $m$ -domain graph we write  $\mathcal{G}_m$  instead of  $\mathcal{G}$ . We also say that  $S_e$  is the set of latent parents in domain  $e$  and denote its cardinality by  $s_e = |S_e|$ . Note that the third condition in Definition 2.1 does not exclude causal relations between the domain-specific latent variables, that is, there may be edges  $v \rightarrow w$  for  $v, w \in I_e$ . Since the sets  $I_e$  satisfy  $I_e \cap I_f = \emptyset$  for  $e \neq f$ , we specify w.l.o.g. the indexing convention  $I_e = \{\ell + 1 + \sum_{i=1}^{e-1} |I_i|, \ldots, \ell + \sum_{i=1}^{e} |I_i|\}$  and  $h = \ell + \sum_{e=1}^{m} |I_e|$ .
Example 2.2. Consider the compact version of a 2-domain graph in Figure 2. There are  $h = 5$  latent nodes where  $\mathcal{L} = \{1,2\}$  are shared and  $I_1 = \{3,4\}$  and  $I_2 = \{5\}$  are domain-specific. A full  $m$ -domain graph is given in Appendix B.
Each  $m$ -domain graph postulates a statistical model that corresponds to the structural equation model in (1) and the mechanisms in (2), with potentially sparse matrices  $A$  and  $G^{e}$ . For two sets of nodes  $W, Y \subseteq \mathcal{H} \cup V$ , we denote by  $D_{WY}$  the subset of edges  $D_{WY} = \{y \to w \in D : w \in W, y \in Y\}$ . Moreover, let  $\mathbb{R}^{D_{WY}}$  be the set of real  $|W| \times |Y|$  matrices  $M = (m_{wy})$  with rows indexed by  $W$  and columns indexed by  $Y$ , such that the support is given by  $D_{WY}$ , that is,  $m_{wy} = 0$  if  $y \to w \notin D_{WY}$ .
Definition 2.3. Let  $\mathcal{G}_m = (\mathcal{H} \cup V, D)$  be an  $m$ -domain graph. Define the map
$$
\phi_ {\mathcal {G} _ {m}}: \mathbb {R} ^ {D _ {V \mathcal {H}}} \times \mathbb {R} ^ {D _ {\mathcal {H} \mathcal {H}}} \longrightarrow \mathbb {R} ^ {| V | \times | \mathcal {H} |}
$$
$$
(G, A) \longmapsto G \cdot (I - A) ^ {- 1}.
$$
Then the multi-domain causal representation (MDCR) model  $\mathcal{M}(\mathcal{G}_m)$  is given by the set of probability measures  $P_{X} = B\# P$ , where  $B \in \mathrm{Im}(\phi_{\mathcal{G}_m})$  and  $P$  is an  $h$ -dimensional probability measure with independent, mean-zero marginals  $P_{i}, i \in \mathcal{H}$ . We say that the pair  $(B,P)$  is a representation of  $P_{X} \in \mathcal{M}(\mathcal{G}_{m})$ .
Definition 2.3 corresponds to the model defined in Equations (1) and (2). If  $P_X \in \mathcal{M}(\mathcal{G}_m)$  with representation  $(B, P)$ , then  $P_X$  is the joint distribution of the observed domains  $X = (X^1, \ldots, X^m)$ . The distribution of the random variables  $\varepsilon_i$  in Equation (1) is given by the marginals  $P_i$ . Moreover, for any matrix  $G \in \mathbb{R}^{D_{V\mathcal{H}}}$ , we denote the submatrix  $G^e = G_{V_e, S_e} \in \mathbb{R}^{d_e \times s_e}$  which coincides with the matrix  $G^e$  from Equation (2). For the graph in Figure 2, we compute a concrete example of the matrix  $B$  in Example B.1 in the Appendix. Importantly, in the rest of the paper we assume to only observe the marginal distribution  $P_{X^e}$  in each domain but not the joint distribution  $P_X$ .
Ultimately, we are interested in recovering the graph  $G_{\mathcal{L}} = (\mathcal{L}, D_{\mathcal{L}\mathcal{L}})$  among the shared latent nodes. We proceed by a two-step approach: In Section 3 we recover the representation  $(B, P)$  of the joint distribution  $P_{X}$ . To be precise, we recover a matrix  $\widehat{B}$  that is equal to  $B$  up to certain permutations of the columns. Then we use the matrix  $\widehat{B}$  to recover the shared latent graph  $G_{\mathcal{L}}$  in Section 4 and show that recovery is possible up to trivial relabeling of latent nodes that appear in the same position of the causal order.
3 Joint Distribution
To identify the joint distribution  $P_{X}$ , we apply identifiability results from linear ICA in each domain separately and match the recovered probability measures  $P_{i}$  for identifying which of them are shared, that is, whether or not  $i \in \mathcal{L}$ . Let  $\mathcal{G}_{m}$  be an  $m$ -domain graph with shared latent nodes  $\mathcal{L}$ , and let  $P_{X} \in \mathcal{M}(\mathcal{G}_{m})$  with representation  $(B, P)$ . Recall that  $B = G(I - A)^{-1}$  with  $G \in \mathbb{R}^{D_{V\mathcal{H}}}$  and  $A \in \mathbb{R}^{D_{\mathcal{H}\mathcal{H}}}$ . We make the following technical assumptions.
(C1) (Different error distributions.) The marginal distributions  $P_{i}, i \in \mathcal{H}$  are non-degenerate, non-symmetric and have unit variance. Moreover, the measures are pairwise different to each other and to the flipped versions, that is,  $P_{i} \neq P_{j}$  and  $P_{i} \neq -P_{j}$  for all  $i, j \in \mathcal{H}$  with  $i \neq j$ . Subsequently, we let  $d$  be a distance on the set of univariate Borel probability measures such that  $d(P_{i}, P_{j}) \neq 0$  and  $d(P_{i}, -P_{j}) \neq 0$  for  $i \neq j$ .
(C2) (Full rank of mixing.) For each  $e \in [m]$ , the matrix  $G^{e} = G_{V_{e}, S_{e}} \in \mathbb{R}^{d_{e} \times s_{e}}$  is of full column rank.