Title: Unpaired Multi-Domain Causal Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: linear structural equation models

causality

representation learning

independent component analysis

structure identifiability

multiple views

graphical model
Abstract: The goal of causal representation learning is to find a representation of data that consists of causally related latent variables. We consider a setup where one has access to data from multiple domains that potentially share a causal representation. Crucially, observations in different domains are assumed to be unpaired, that is, we only observe the marginal distribution in each domain but not their joint distribution. In this paper, we give sufficient conditions for identifiability of the joint distribution and the shared causal graph in a linear setup. Identifiability holds if we can uniquely recover the joint distribution and the shared causal representation from the marginal distributions in each domain. We transform our results into a practical method to recover the shared latent causal graph.

Unpaired Multi-Domain Causal Representation Learning

Abstract

The goal of causal representation learning is to find a representation of data that consists of causally related latent variables. We consider a setup where one has access to data from multiple domains that potentially share a causal representation. Crucially, observations in different domains are assumed to be unpaired, that is, we only observe the marginal distribution in each domain but not their joint distribution. In this paper, we give sufficient conditions for identifiability of the joint distribution and the shared causal graph in a linear setup. Identifiability holds if we can uniquely recover the joint distribution and the shared causal representation from the marginal distributions in each domain. We transform our results into a practical method to recover the shared latent causal graph.

1 Introduction

An important challenge in machine learning is the integration and translation of data across multiple domains (Zhu et al., 2017; Zhuang et al., 2021). Researchers often have access to large amounts of unpaired data from several domains, e.g., images and text. It is then desirable to learn a probabilistic coupling between the observed marginal distributions that captures the relationship between the domains. One approach to tackle this problem is to assume that there is a latent representation that is invariant across the different domains (Bengio et al., 2013; Ericsson et al., 2022). Finding a probabilistic coupling then boils down to learning such a latent representation, that is, learning high-level, latent variables that explain the variation of the data within each domain as well as similarities across domains.

In traditional representation learning, the latent variables are assumed to be statistically independent, see for example the literature on independent component analysis (Hyvärinen and Oja, 2000; Comon and Jutten, 2010; Khemakhem et al., 2020). However, the assumption of independence can be too stringent and a poor match to reality. For example, the presence of clouds and the presence of wet roads in an image may be dependent, since clouds may cause rain which may in turn cause wet roads. Thus, it is natural to seek a causal representation , that is, a set of high-level causal variables and relations among them (Schölkopf et al., 2021; Yang et al., 2021b). Figure 1 illustrates the setup of multi-domain causal representation learning, where multiple domains provide different views on a shared causal representation.

Our motivation to study multi-domain causal representations comes, in particular, from single-cell data in biology. Given a population of cells, different technologies such as imaging and sequencing

provide different views on the population. Crucially, since these technologies destroy the cells, the observations are uncoupled, i.e., a specific cell may either be used for imaging or sequencing but not both. The aim is to integrate the different views on the population to study the underlying causal mechanisms determining the observed features in various domains (Butler et al., 2018; Stuart et al., 2019; Liu et al., 2019; Yang et al., 2021a; Lopez et al., 2022; Gossi et al., 2023; Cao et al., 2022). Unpaired multi-domain data also appears in many applications other than single-cell biology. For example, images of similar objects are captured in different environments (Beery et al., 2018), large biomedical and neuroimaging data sets are collected in different domains (Miller et al., 2016; Essen et al., 2013; Shafto et al., 2014; Wang et al., 2003), or stocks are traded in different markets.

In this paper, we study identifiability of the shared causal representation, that is, its uniqueness in the infinite data limit. Taking on the same perspective as, for example, in Schölkopf et al. (2021) and Squires et al. (2023), we assume that observed data is generated in two steps. First, the latent variables Z = ( Z i ) i ∈H are sampled from a distribution P Z , where P Z is determined by an unknown structural causal model among the latent variables. Then, in each domain e ∈ { 1 , . . . , m } , the observed vector X e ∈ R d e is the image of a subset of the latent variables under a domain-specific, injective mixing function g e . That is,

where S e ⊆ H is a subset of indices. A priori, it is unknown whether a latent variable Z i with i ∈ S e is shared across domains or domain-specific. Even the number of latent variables which are shared across domains is unknown. Moreover, we only observe the marginal distribution of each random vector X e , but none of the joint distributions over pairs X e , X f for e = f . Said differently, observations across domains are unpaired. Assuming that the structural causal model among the latent variables as well as the mixing functions are linear, our main contributions are:

̸

We lay out conditions under which we can identify the joint distribution of X 1 , . . . , X m .

We give additional conditions under which we are able to identify the causal structure among the shared latent variables.

In particular, identifiability of the joint distribution across domains enables data translation. That is, given observation x in domain e , translation to domain f can be achieved by computing E [ X f | X e = x ] . Furthermore, identifying the causal structure among the shared latent variables lets us study the effect of interventions on the different domains.

The main challenge in proving rigorous identifiability results for multi-domain data is that we cannot apply existing results for single-domain data in each domain separately. Even if the causal structure of the latent variables in a single domain is identifiable, it remains unclear how to combine multiple causal structures, i.e., in which way latent variables are shared. We circumvent this problem via a two-step approach: First, we extend the identifiability of linear independent component analysis (Comon, 1994; Hyvärinen and Oja, 2000; Eriksson and Koivunen, 2004; Mesters and Zwiernik, 2022) to the multi-domain setup, which allows us to identify the joint distribution and distinguish between shared and domain-specific latent variables. Moreover, we identify an 'overall mixing matrix' and, in a second step, exploit sparsity constraints in this matrix to identify the causal structure among the shared latent variables. This leverages recent results on causal discovery under measurement error in single domains that also exploit sparsity (Xie et al., 2020; Chen et al., 2022; Xie et al., 2022; Huang et al., 2022). Although we emphasize that our focus in this paper is on identifiability, our proofs also suggest methods to learn the joint distribution as well as the shared causal graph from finite samples. We provide algorithms for the noisy setting and, moreover, we analyze how the number of domains reduce uncertainty with respect to the learned representation.

The paper is organized as follows. In the next paragraphs we discuss further related work. Section 2 provides a precise definition of the considered setup. In Section 3 we consider identifiability of the joint distribution. Using these results, we study identifiability of the causal graph in Section 4. We conclude with a small simulation study as a proof of concept for the finite sample setting in Section 5. Due to space constraints, the detailed discussion of the finite sample setting is deferred to the Appendix. Moreover, the Appendix contains all proofs, discussions on the necessity of our assumptions, and additional examples and simulation results.

Multi-domain Integration. Motivated by technological developments for measuring different modalities at single-cell resolution, several methods have been proposed recently for domain

Z1

Z4

Z3

Z6

X1

Z7

↑

X3

T

X2

画

translation between unpaired data. The proposed methods rely on a variety of techniques, including manifold alignment (Welch et al., 2017; Amodio and Krishnaswamy, 2018; Liu et al., 2019), matrix factorization (Duren et al., 2018), correlation analysis (Barkas et al., 2019; Stuart et al., 2019), coupled autoencoders (Yang and Uhler, 2019), optimal transport (Cao et al., 2022), regression analysis (Yuan and Duren, 2022), and semisupervised learning (Lin et al., 2022). Implicitly, these methods presume the existence of a shared latent space where the different modalities either completely align or at least overlap. However, to the best of our knowledge, none of these methods have rigorous identifiability guarantees, i.e., the methods are not guaranteed to recover a correct domain translation mapping even for infinite data. Our work advances the theoretical understanding of multi-domain integration by providing identifiability guarantees on recovering the shared latent space.

Group Independent Component Analysis. The primary tool that we use for identifiability is linear independent component analysis (ICA) (Comon, 1994; Eriksson and Koivunen, 2004). Many works extend ICA to the multi-domain setting. These methods primarily come from computational neuroscience, where different domains correspond to different subjects or studies. However, to the best of our knowledge, all prior works require pairing between samples. These works can be categorized based on whether the samples are assumed to be voxels (Calhoun et al., 2001; Esposito et al., 2005), time points (Svensén et al., 2002; Varoquaux et al., 2009; Hyvärinen and Ramkumar, 2013), or either (Beckmann and Smith, 2005; Sui et al., 2009). For reviews, see Calhoun et al. (2009) and Chabriel et al. (2014). Related are methods for independent vector analysis (Kim et al., 2006; Anderson et al., 2014; Bhinge et al., 2019) and multiset canonical correlation analysis (Nielsen, 2002; Li et al., 2011; Klami et al., 2014), which allow the latent variables to take on different values in each domain but still require sample pairing. Most of the mentioned methods lack identifiability guarantees, only newer work (Richard et al., 2021) provides sufficient conditions for identifiability. Furthermore, all mentioned methods assume that every latent variable is shared across all domains, while our setup allows for shared and domain-specific latent variables. Some methods, e.g., Lukic et al. (2002), Maneshi et al. (2016), and Pandeva and Forré (2023), permit both shared and domain-specific components, but only consider the paired setting. In this paper, we extend these results to the unpaired setting.

Latent Structure Discovery. Learning causal structure between latent variables has a long history, e.g., in measurement models (Silva et al., 2006). One recent line of work studies the problem under the assumption of access to interventional data (e.g., Liu et al., 2022; Squires et al., 2023). In particular, Squires et al. (2023) show that the latent graph is identifiable if the interventions are sufficiently diverse. Another line of work, closer to ours and not based on interventional data, shows that the graph is identified under certain sparsity assumptions on the mixing functions (Xie et al., 2020; Chen et al., 2022; Xie et al., 2022; Huang et al., 2022). However, these methods are not suitable in our setup since they require paired data in a single domain. One cannot apply them in each domain separately since it would be unclear how to combine the multiple latent causal graphs, that is, which of the latent variables are shared. In this work, we lay out sparsity assumptions on the mixing

functions that are tailored to the unpaired multi-domain setup. The works of Adams et al. (2021) and Zeng et al. (2021) may be considered closest to ours as they also treat a setting with multiple domains and unpaired data. However, our setup and results are more general. Adams et al. (2021) assume that the number of observed variables are the same in each domain, whereas we consider domains of different dimensions corresponding to the fact that observations may be of very different nature. Further, we allow for shared and domain-specific latent variables, where the number of shared latent variables is unknown, while in Adams et al. (2021) it is assumed that all latent variables are shared. Compared to Zeng et al. (2021), we consider a general but fixed number of observed variables, while Zeng et al. (2021) only show identifiability of the full model in a setup where the number of observed variables in each domain increases to infinity. On a more technical level, the conditions in Zeng et al. (2021) require two pure children to identify the shared latent graph, while we prove identifiability under the weaker assumption of two partial pure children; see Section 4 for precise definitions.

Notation. Let N be the set of nonnegative integers. For positive n ∈ N , we define [ n ] = { 1 , . . . , n } . For a matrix M ∈ R a × b , we denote by M A,B the submatrix containing the rows indexed by A ⊆ [ a ] and the columns indexed by B ⊆ [ b ] . Moreover, we write M B for the submatrix containing all rows but only the subset of columns indexed by B . Similarly, for a tuple x = ( x 1 , . . . , x b ) , we denote by x B the tuple only containing the entries indexed by B . A matrix Q = Q σ ∈ R p × p is a signed permutation matrix if it can be written as the product of a diagonal matrix D with entries D ii ∈ {± 1 } and a permutation matrix ˜ Q σ with entries ( ˜ Q σ ) ij = ✶ j = σ ( i ) , where σ is a permutation on p elements. Let P be a p -dimensional joint probability measure of a collection of random variables Y 1 , . . . , Y p . Then we denote by P i the marginal probability measure such that Y i ∼ P i . We say that P has independent marginals if the random variables Y i are mutually independent. Moreover, we denote by M # P the d -dimensional push-forward measure under the linear map defined by the matrix M ∈ R d × p . If Q is a signed permutation matrix and the probability measure P has independent marginals, then Q # P also has independent marginals. For univariate probability measures we use the shorthand ( -1)# P = -P .

2 Setup

Let H = [ h ] for h ≥ 1 , and let Z = ( Z 1 , . . . , Z h ) be latent random variables that follow a linear structural equation model. That is, the variables are related by a linear equation

with h × h parameter matrix A = ( a ij ) and zero-mean, independent random variables ε = ( ε 1 , . . . , ε h ) that represent stochastic errors. Assume that we have observed random vectors X e ∈ R d e in multiple domains of interest e ∈ [ m ] , where the dimension d e may vary across domains. Each random vector is the image under a linear function of a subset of the latent variables. In particular, we assume that there is a subset L ⊆ H representing the shared latent space such that each X e is generated via the mechanism

where I e ⊆ H \ L . We say that the latent variable Z I e are domain-specific for domain e ∈ [ m ] while the latent variables Z L are shared across all domains. As already noted, we are motivated by settings where the shared latent variables Z L capture the key causal relations and the different domains are able to give us combined information about these relations. Likewise, we may think about the domainspecific latent variables Z I e as 'noise' in each domain, independent of the shared latent variables. Specific models are now derived from (1)-(2) by assuming specific (but unknown) sparsity patterns in A and G e . Each model is given by a 'large' directed acyclic graph (DAG) that encodes the multi-domain setup. To formalize this, we introduce pairwise disjoint index sets V 1 , . . . , V m , where V e indexes the coordinates of X e , i.e., X e = ( X v : v ∈ V e ) and | V e | = d e . Then V = V 1 ∪ · · · ∪ V m indexes all observed random variables. We define an m -domain graph such that the latent nodes are the only parents of observed nodes and there are no edges between shared and domain-specific latent nodes.

Definition 2.1. Let G be a DAG whose node set is the disjoint union H∪ V = H∪ V 1 ∪ · · · ∪ V m . Let D be the edge set of G . Then G is an m -domain graph with shared latent nodes L = [ ℓ ] ⊆ H if the following is satisfied:

All parent sets contain only latent variables, i.e., pa ( v ) = { w : w → v ∈ D } ⊆ H for all v ∈ H ∪ V .

3

4

1

2

5

V

1

V

2

̸

The set L consists of the common parents of variables in all different domains, i.e., u ∈ L if and only if u ∈ pa ( v ) ∩ pa ( w ) for v ∈ V e , w ∈ V f with e = f .

̸

Let I e = S e \ L be the domain-specific latent nodes, where S e := pa ( V e ) = ∪ v ∈ V e pa ( v ) ⊆ H . Then there are no edges in D that connect a node in L and a node ∪ m e =1 I e or that connect a node in I e and a node in I f for any e = f .

̸

To emphasize that a given DAG is an m -domain graph we write G m instead of G . We also say that S e is the set of latent parents in domain e and denote its cardinality by s e = | S e | . Note that the third condition in Definition 2.1 does not exclude causal relations between the domain-specific latent variables, that is, there may be edges v → w for v, w ∈ I e . Since the sets I e satisfy I e ∩ I f = ∅ for e = f , we specify w.l.o.g. the indexing convention I e = { ℓ +1+ ∑ e -1 i =1 | I i | , . . . , ℓ + ∑ e i =1 | I i |} and h = ℓ + ∑ m e =1 | I e | .

Example 2.2. Consider the compact version of a 2 -domain graph in Figure 2. There are h = 5 latent nodes where L = { 1 , 2 } are shared and I 1 = { 3 , 4 } and I 2 = { 5 } are domain-specific. A full m -domain graph is given in Appendix B.

Each m -domain graph postulates a statistical model that corresponds to the structural equation model in (1) and the mechanisms in (2), with potentially sparse matrices A and G e . For two sets of nodes W,Y ⊆ H ∪ V , we denote by D WY the subset of edges D WY = { y → w ∈ D : w ∈ W,y ∈ Y } . Moreover, let R D WY be the set of real | W | × | Y | matrices M = ( m wy ) with rows indexed by W and columns indexed by Y , such that the support is given by D WY , that is, m wy = 0 if y → w ̸∈ D WY .

Definition 2.3. Let G m = ( H∪ V, D ) be an m -domain graph. Define the map

Then the multi-domain causal representation (MDCR) model M ( G m ) is given by the set of probability measures P X = B # P , where B ∈ Im ( ϕ G m ) and P is an h -dimensional probability measure with independent, mean-zero marginals P i , i ∈ H . We say that the pair ( B,P ) is a representation of P X ∈ M ( G m ) .

Definition 2.3 corresponds to the model defined in Equations (1) and (2). If P X ∈ M ( G m ) with representation ( B,P ) , then P X is the joint distribution of the observed domains X = ( X 1 , . . . , X m ) . The distribution of the random variables ε i in Equation (1) is given by the marginals P i . Moreover, for any matrix G ∈ R D V H , we denote the submatrix G e = G V e ,S e ∈ R d e × s e which coincides with the matrix G e from Equation (2). For the graph in Figure 2, we compute a concrete example of the matrix B in Example B.1 in the Appendix. Importantly, in the rest of the paper we assume to only observe the marginal distribution P X e in each domain but not the joint distribution P X .

Ultimately, we are interested in recovering the graph G L = ( L , D LL ) among the shared latent nodes. We proceed by a two-step approach: In Section 3 we recover the representation ( B,P ) of the joint distribution P X . To be precise, we recover a matrix ̂ B that is equal to B up to certain permutations of the columns. Then we use the matrix ̂ B to recover the shared latent graph G L in Section 4 and show that recovery is possible up to trivial relabeling of latent nodes that appear in the same position of the causal order.

3 Joint Distribution

To identify the joint distribution P X , we apply identifiability results from linear ICA in each domain separately and match the recovered probability measures P i for identifying which of them are shared, that is, whether or not i ∈ L . Let G m be an m -domain graph with shared latent nodes L , and let P X ∈ M ( G m ) with representation ( B,P ) . Recall that B = G ( I -A ) -1 with G ∈ R D V H and A ∈ R D HH . We make the following technical assumptions.

̸

̸

(C1) (Different error distributions.) The marginal distributions P i , i ∈ H are non-degenerate, non-symmetric and have unit variance. Moreover, the measures are pairwise different to each other and to the flipped versions, that is, P i = P j and P i = -P j for all i, j ∈ H with i = j . Subsequently, we let d be a distance on the set of univariate Borel probability measures such that d ( P i , P j ) = 0 and d ( P i , -P j ) = 0 for i = j .

̸

̸

̸

(C2) (Full rank of mixing.) For each e ∈ [ m ] , the matrix G e = G V e ,S e ∈ R d e × s e is of full column rank.

By not allowing symmetric distributions in Condition (C1), we assume in particular that the distributions of the errors are non-Gaussian. Non-Gaussianity together with the assumptions of pairwise different and non-symmetric error distributions allow us to extend the results on identifiability of linear ICA to the unpaired multi-domain setup and to identify the joint distribution. In particular, the assumption of pairwise different error distributions allows for 'matching' the distributions across domains to identify the ones corresponding to the shared latent space. Non-symmetry accounts for the sign-indeterminacy of linear ICA when matching the distributions. We discuss the necessity of these assumptions in Remark 3.2 and, in more detail, in Appendix C. Note that Condition (C1) is always satisfied in a generic sense, that is, randomly chosen probability distributions on the real line are pairwise different and non-symmetric with probability one. Finally, Condition (C2) requires in particular that for each shared latent node k ∈ L there is at least one node v ∈ V e in every domain e ∈ [ m ] such that k ∈ pa ( v ) .

Under Conditions (C1) and (C2) we are able to derive a sufficient condition for identifiability of the joint distribution. Let SP ( p ) be the set of p × p signed permutation matrices. We define the set of signed permutation block matrices :

Our main result is the following.

Theorem 3.1. Let G m be an m -domain graph with shared latent nodes L = [ ℓ ] , and let P X ∈ M ( G m ) with representation ( B,P ) . Suppose that m ≥ 2 and that Conditions (C1) and (C2) are satisfied. Let ( ̂ ℓ, ̂ B, ̂ P ) be the output of Algorithm 1. Then ̂ ℓ = ℓ and

for a signed permutation block matrix Ψ ∈ Π .

Theorem 3.1 says that the matrix B is identifiable up to signed block permutations of the columns. Under the assumptions of Theorem 3.1 it holds that ̂ B # ̂ P is equal to P X . That is, the joint distribution of the domains is identifiable.

Remark 3.2 . While Theorem 3.1 is a sufficient condition for identifiability of the joint distribution, we emphasize that pairwise different error distributions are in most cases also necessary; we state the exact necessary condition in Proposition C.1 in the Appendix. Said differently, if one is willing to assume that conceptually different latent variables also follow a different distribution, then identification of these variables is possible, and otherwise (in most cases) not. Apart from pairwise different error distributions, non-symmetry is then required to fully identify the joint distribution whose dependency structure is determined by the shared latent variables. If the additional assumption on non-symmetry is not satisfied, then it is still possible to identify the shared, conceptually different latent variables, which becomes clear by inspecting the proofs of Theorem 3.1 and Proposition C.1. The non-identifiability of the joint distribution would only result in sign indeterminacy, that is, entries of the matrix ̂ B could have a flipped sign.

Remark 3.3 . By checking the proof of Theorem 3.1, the careful reader may notice that the statement of the theorem still holds true when we relax the third condition in the definition of an m -domain

̸

Algorithm 1 IdentifyJointDistribution

1: Input: Probability measures P X e for all e ∈ [ m ] .

2: Output: Number of shared latent variables ̂ ℓ , matrix ̂ B and probability measure ̂ P with independent marginals.

3: for e ∈ [ m ] do

4: Linear ICA: Find the smallest value ̂ s e such that P X e = ̂ B e # P e for a matrix ̂ B e ∈ R d e × ̂ s e and an ̂ s e -dimensional probability measure P e with independent, mean-zero and unit-variance marginals P e i .

5: end for

6: Matching: Let ̂ ℓ be the maximal number such that there are signed permutation matrices { Q e } e ∈ [ m ] satisfying

̸

for all i = 1 , . . . , ̂ ℓ and for all f = e . Let ̂ L = { 1 , . . . , ̂ ℓ } .

7: Construct the matrix ̂ B and the tuple of probability measures ̂ P given by

8: return ( ̂ ℓ , ̂ B , ̂ P ).

graph. That is, one may allow directed paths from shared to domain-specific latent nodes but not vice versa. For example, an additional edge 1 → 4 between the latent nodes 1 and 4 would be allowed in the graph in Figure 2. In this case, the dependency structure of the domains is still determined by the shared latent space. However, the structural assumption that there are no edges between shared and domain-specific latent nodes is made for identifiability of the shared latent graph in Section 4.

Remark 3.4 . The computational complexity of Algorithm 1 depends on the complexity of the chosen linear ICA algorithm, to which we make m calls. Otherwise, the dominant part is the matching in Line 6 with worst case complexity O ( m · max e ∈ [ m ] d 2 e ) , where we recall that m is the number of domains and d e is the dimension of domain e .

In Appendix D we state a complete version of Algorithm 1 for the finite sample setting. In particular, we provide a method for the matching in Line 6 based on the two-sample Kolmogorov-Smirnov test. For finite samples, there might occur false discoveries, that is, distributions are matched that are actually not the same. With our method, we show that the probability of falsely discovering shared nodes shrinks exponentially with the number of domains.

4 Identifiability of the Causal Graph

We return to our goal of identifying the causal graph G L = ( L , D LL ) among the shared latent nodes. By Theorem 3.1, we can identify the representation ( B,P ) of P X ∈ M ( G m ) from the marginal distributions. In particular, we recover the matrix ̂ B = B Ψ for a signed permutation block matrix Ψ ∈ Π . Moreover, we know which columns correspond to the shared latent nodes. That is, we know that the submatrix ̂ B L obtained by only considering the columns indexed by L = ̂ L = [ ℓ ] is equal to B L Ψ L , where Ψ L ∈ SP ( ℓ ) .

Problem 4.1. Let B ∈ Im ( ϕ G m ) for an m -domain graph G m with shared latent nodes L . Given ̂ B L = B L Ψ L with Ψ L a signed permutation matrix, when is it possible to identify the graph G L ?

Recently, Xie et al. (2022) and Dai et al. (2022) show that, in the one-domain setting with independent additive noise, the latent graph can be identified if each latent variable has at least two pure children. We obtain a comparable result tailored to the multi-domain setup.

Definition 4.2. Let G m = ( H∪ V, D ) be an m -domain graph with shared latent nodes L ⊆ H . For k ∈ L , we say that an observed node v ∈ V is a partial pure child of k if pa ( v ) ∩ L = { k } .

.

Algorithm 2 IdentifySharedGraph

1: Input: Matrix B ∗ ∈ R | V |× ℓ .

2: Output: Parameter matrix ̂ A ∈ R ℓ × ℓ .

3: Remove rows B ∗ i, L from the matrix B ∗ that are completely zero.

̸

4: Find tuples ( i k , j k ) k ∈L with i k = j k such that

(i) rank ( B ∗ { i k ,j k } , L ) = 1 for all k ∈ L and

(ii) rank ( B ∗ { i k ,i q } , L ) = 2 for all k, q ∈ L with k = q .

5: Let I = { i 1 , . . . , i ℓ } and consider B ∗ I, L ∈ R ℓ × ℓ .

̸

6: Find two permutation matrices R 1 and R 2 such that W = R 1 B ∗ I, L R 2 is lower triangular.

7: Multiply each column of W by the sign of its corresponding diagonal element. This yields a new matrix ˜ W with all diagonal elements positive.

8: Divide each row of ˜ W by its corresponding diagonal element. This yields a new matrix ˜ W ′ with all diagonal elements equal to one.

9: Compute ̂ A = I -( ˜ W ′ ) -1 .

10: return ̂ A .

For a partial pure child v ∈ V , there may still be domain-specific latent nodes that are parents of v . Definition 4.2 only requires that there is exactly one parent that is in the set L . This explains the name partial pure child; see Example B.2 in the Appendix for further elaboration.

W.l.o.g. we assume in this section that the shared latent nodes are topologically ordered such that i → j ∈ D LL implies i < j for all i, j ∈ L . We further assume:

(C3) (Two partial pure children across domains.) For each shared latent node k ∈ L , there exist two partial pure children.

(C4) (Rank faithfulness.) For any two subsets Y ⊆ V and W ⊆ L , we assume that

The two partial pure children required in Condition (C3) may either be in distinct domains or in a single domain. This is a sparsity condition on the large mixing matrix G . In Appendix C we discuss that the identification of the joint latent graph is impossible without any sparsity assumptions. We conjecture that two partial pure children are not necessary, but we leave it open for future work to find a non-trivial necessary condition. Roughly speaking, we assume in Condition (C4) that no configuration of edge parameters coincidentally yields low rank. The set of matrices B ∈ Im ( ϕ G m ) that violates (C4) is a subset of measure zero of Im ( ϕ G m ) with respect to the Lebesgue measure. Note that our conditions do not impose constraints on the graph G L . Our main tool to tackle Problem 4.1 will be the following lemma.

Lemma 4.3. Let B ∈ Im ( ϕ G m ) for an m -domain graph G m . Suppose that Condition (C4) is satisfied and that there are no zero-rows in B L . Let v, w ∈ V . Then rank ( B { v,w } , L ) = 1 if and only if there is a node k ∈ L such that both v and w are partial pure children of k .

The condition on no zero-rows in Lemma 4.3 is needed since we always have rank ( B { v,w } , L ) ≤ 1 if one of the two rows is zero. However, this is no additional structural assumption since we allow zero-rows when identifying the latent graph; c.f. Algorithm 2. The lemma allows us to find partial pure children by testing ranks on the matrix ̂ B L . If ( i 1 , j 1 ) and ( i 2 , j 2 ) are partial pure children of two nodes in L , we make sure that these two nodes are different by checking that rank ( B { i 1 ,i 2 } , L ) = 2 .

For a DAG G = ( V, D ) , we define S ( G ) to be the set of permutations on | V | elements that are consistent with the DAG, i.e., σ ∈ S ( G ) if and only if σ ( i ) < σ ( j ) for all edges i → j ∈ D . The following result is the main result of this section.

Theorem 4.4. Let ̂ B = B Ψ with B ∈ Im ( ϕ G m ) and Ψ ∈ Π , and define B ∗ = ̂ B L to be the input of Algorithm 2. Assume that Conditions (C3) and (C4) are satisfied, and let ̂ A be the output of Algorithm 2. Then ̂ A = Q ⊤ σ A L , L Q σ for a signed permutation matrix Q σ with σ ∈ S ( G L ) . Moreover, if G vk > 0 for G ∈ R D V H whenever v is a pure child of k , then Q σ is a permutation matrix.

̸

Theorem 4.4 says that the graph G L can be recovered up to a permutation of the nodes that preserves the property that i → j implies i < j ; see Remark 4.5. Since the columns of the matrix ̂ B are not only permuted but also of different signs, we solve the sign indeterminacy column-wise in Line 7 before removing the scaling indeterminacy row-wise in Line 8. In case the coefficients of partial pure children are positive, this ensures that Q σ is a permutation matrix and we have no sign indeterminacy. In Appendix D we adapt Algorithm 2 for the empirical data setting, where we only have ̂ B L ≈ B L ψ L . Remark 4.5 . Let ̂ A be the output of Alg. 2. Then we construct the graph ̂ G L = ( L , ̂ D LL ) as the graph with edges j → i ∈ ̂ D LL if and only if ̂ A ij = 0 . Condition (C4) ensures that ̂ G L is equivalent to G L in the sense that there is a permutation σ ∈ S ( G L ) such that ̂ D LL = { σ ( i ) → σ ( j ) : i → j ∈ D LL } .

Example 4.6. As highlighted in the introduction, the unpaired multi-domain setup is motivated by applications from single-cell biology. For example, consider the domains of (i) gene expression and (ii) high-level phenotypic features extracted from imaging assays (e.g. McQuin et al., 2018). We argue that the requirement of two partial pure children is justifiable on such data as follows. The condition requires, for example, that for each shared latent variable, (i) the expression of some gene depends only upon that shared latent variable plus domain-specific latent variables, and (ii) one of the high-level phenotypic features depends only on the same latent feature plus domain-specific latent variables. Many genes have highly specialized functions, so (i) is realistic, and similarly many phenotypic features are primarily controlled by specific pathways, so (ii) is justified.

Remark 4.7 . In Algorithm 2, we determine the rank of a matrix by Singular Value Decomposition, which has worst case complexity O ( mn min { n, m } ) for an m × n matrix. Since Line 4 is the dominant part, we conclude that the worst case complexity of Algorithm 2 is O ( | V | 2 · ℓ ) .

5 Simulations

In this section we report on a small simulation study to illustrate the validity of our adapted algorithms for finite samples (detailed in Appendix D). We emphasize that this should only serve as a proof of concept as the focus of our work lies on identifiability. In future work one may develop more sophisticated methods; c.f. Appendix G. The adapted algorithms have a hyperparameter γ , which is a threshold on singular values to determine the rank of a matrix. In our simulations we use γ = 0 . 2 .

Data Generation. In each experiment we generate 1000 random models with ℓ = 3 shared latent nodes. We consider different numbers of domains m ∈ { 2 , 3 } and assume that there are | I e | = 2 domain-specific latent nodes for each domain. The dimensions are given by d e = d/m for all e ∈ [ m ] and d = 30 . We sample the m -domain graph G m on the shared latent nodes as follows. First, we sample the graph G L from an Erd˝ os-Rényi model with edge probability 0 . 75 and assume that there are no edges between other latent nodes, that is, between L and H\L and within H\L . Then we fix two partial pure children for each shared latent node k ∈ L and collect them in the set W . The remaining edges from L to V \ W and from H to V are sampled from an Erd˝ os-Rényi model with edge probability 0 . 9 . Finally, the (nonzero) entries of G and A are sampled from Unif ( ± [0 . 25 , 1]) . The distributions of the error variables are specified in Appendix E. For simplicity, we assume that the sample sizes coincide, that is, n e = n for all e ∈ [ m ] , and consider n ∈ { 1000 , 2500 , 5000 , 10000 , 25000 } .

Results. First, we plot the average number of shared nodes ̂ ℓ in our experiments in Figure 3 (a). Especially for low sample sizes, we see that fewer shared nodes are detected with more domains. However, by inspecting the error bars we also see that the probability of detecting too many nodes ̂ ℓ > ℓ decreases drastically when considering 3 instead of 2 domains. This suggests that the number of falsely detected shared nodes is very low, as expected by Theorem D.3. Our findings show that more domains lead to a more conservative discovery of shared nodes, but whenever a shared node is determined this is more certain. Moreover, we measure the error in estimating ̂ B ̂ L in Figure 3 (b), that is, the error in the 'shared' columns. We take

where ∥ · ∥ F denotes the Frobenius norm and β ℓ, ̂ ℓ = min { ℓ, ̂ ℓ } · ∑ m e =1 d e denotes the number of entries of the matrix over which the norm is taken. In the cases ℓ = ̂ ℓ , we also measure the

(a)

(b)

(c)

3.5

0.05

1.4

3.0

2domains

1.2

2domains

2.5

scoreB

0.04

3domains

Median scoreA

1.0

3domains

of

Average

2.0

0.03

0.8

1.5

ian

0.02

0.6

Media

1.0

2domains

0.4

0.5

3domains

0.01

0.2

0.0

0.00

0.0

10

3

10

4

10°

3

10

4

10

3

10

4

Samplesize

Sample size

Samplesize

performance of recovering the shared latent graph G L in Figure 3 (c) by taking

As expected, the median estimation errors for B L and A L , L decrease with increasing sample size. In Appendix F we provide additional simulations with larger ℓ . Moreover, we consider setups where we violate specific assumptions, such as pairwise different distributions (C1) and two partial pure children (C3). The results emphasize that the conditions are necessary for the algorithms provided. The computations were performed on a single thread of an Intel Xeon Gold 6242 R processor ( 3 . 1 GHz), with a total computation time of 12 hours for all simulations presented in this paper (including Appendix).

6 Discussion

This work introduces the problem of causal representation learning from unpaired multi-domain observations, in which multiple domains provide complementary information about a set of shared latent nodes that are the causal quantities of primary interest. For this problem, we laid out a setting in which we can provably identify the causal relations among the shared latent nodes. To identify the desired causal structure, we proposed a two-step approach where we first make use of linear ICA in each domain separately and match the recovered error distributions to identify shared nodes and the joint distribution of the domains. In the second step, we identify the causal structure among the shared latent variables by testing rank deficiencies in the 'overall mixing matrix' B . To the best of our knowledge, our guarantees are the first principled identifiability results for shared causal representations in a general, unpaired multi-domain setting.

We proposed algorithms for recovering the joint distribution and the shared latent space making our proofs constructive. While our focus is on identifiability guarantees, we show in Appendix D how our proofs give rise to algorithms for the finite sample setting. Moreover, we propose a method to match approximate error distributions and show that the probability of falsely discovering shared nodes decreases exponentially in the number of domains. Our work opens up numerous directions for future work as we discuss in Appendix G.

Acknowledgments and Disclosure of Funding

This project was initiated while the first author was a visitor at the Eric and Wendy Schmidt Center of the Broad Institute of MIT and Harvard. The project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 883818), NCCIH/NIH (1DP2AT012345), ONR (N00014-22-1-2116), DOE-ASCR (DE-SC0023187), NSF (DMS-1651995), the MIT-IBM Watson AI Lab, and a Simons Investigator Award. Nils Sturma acknowledges support by the Munich Data Science Institute at the Technical University of Munich via the Linde/MDSI PhD Fellowship program. Chandler Squires was partially supported by an NSF Graduate Research Fellowship.

References

Adams, J., Hansen, N., and Zhang, K. (2021). Identification of partially observed linear causal models: Graphical conditions for the non-gaussian and heterogeneous cases. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 22822-22833. Curran Associates, Inc.

Ahuja, K., Mahajan, D., Wang, Y., and Bengio, Y. (2023). Interventional causal representation learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 372-407. PMLR.

Amodio, M. and Krishnaswamy, S. (2018). MAGAN: Aligning biological manifolds. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 215-223. PMLR.

Anderson, M., Fu, G.-S., Phlypo, R., and Adalı, T. (2014). Independent vector analysis: Identification conditions and performance bounds. IEEE Trans. Signal Process. , 62(17):4399-4410.

Bach, F. R. and Jordan, M. I. (2003). Kernel independent component analysis. J. Mach. Learn. Res. , 3(1):1-48.

Barkas, N., Petukhov, V., Nikolaeva, D., Lozinsky, Y., Demharter, S., Khodosevich, K., and Kharchenko, P. V. (2019). Joint analysis of heterogeneous single-cell RNA-seq dataset collections. Nat. Methods , 16(8):695-698.

Beckmann, C. F. and Smith, S. M. (2005). Tensorial extensions of independent component analysis for multisubject fMRI analysis. Neuroimage , 25(1):294-311.

Beery, S., Van Horn, G., and Perona, P. (2018). Recognition in terra incognita. In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y., editors, Computer Vision - ECCV 2018 , pages 472-489, Cham. Springer International Publishing.

Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. , 35(8):1798-1828.

Bhinge, S., Mowakeaa, R., Calhoun, V. D., and Adalı, T. (2019). Extraction of time-varying spatiotemporal networks using parameter-tuned constrained IVA. IEEE Trans. Med. Imaging , 38(7):1715-1725.

Buchholz, S., Besserve, M., and Schölkopf, B. (2022). Function classes for identifiable nonlinear independent component analysis. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 16946-16961. Curran Associates, Inc.

Butler, A., Hoffman, P., Smibert, P., Papalexi, E., and Satija, R. (2018). Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nat. Biotechnol. , 36(5):411-420.

Calhoun, V. D., Adali, T., Pearlson, G. D., and Pekar, J. J. (2001). A method for making group inferences from functional MRI data using independent component analysis. Hum. Brain Mapp. , 14(3):140-151.

Calhoun, V. D., Liu, J., and Adalı, T. (2009). A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data. Neuroimage , 45(1):163-172.

Cao, K., Gong, Q., Hong, Y., and Wan, L. (2022). A unified computational framework for single-cell data integration with optimal transport. Nat. Comm. , 13(1).

Cardoso, J. and Souloumiac, A. (1993). Blind beamforming for non-gaussian signals. IEE Proceedings F Radar and Signal Processing , 140(6):362.

Chabriel, G., Kleinsteuber, M., Moreau, E., Shen, H., Tichavsky, P., and Yeredor, A. (2014). Joint matrices decompositions and blind source separation: A survey of methods, identification, and applications. IEEE Signal Process. Mag. , 31(3):34-43.

Chen, Z., Xie, F., Qiao, J., Hao, Z., Zhang, K., and Cai, R. (2022). Identification of linear latent variable model with arbitrary distribution. Proceedings of the AAAI Conference on Artificial Intelligence , 36(6):6350-6357.

Comon, P. (1994). Independent component analysis, a new concept? Signal Process. , 36(3):287-314.

Comon, P. and Jutten, C. (2010). Handbook of Blind Source Separation: Independent Component Analysis and Applications . Elsevier.

Dai, H., Spirtes, P., and Zhang, K. (2022). Independence testing-based approach to causal discovery under measurement error and linear non-gaussian models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 27524-27536. Curran Associates, Inc.

Duren, Z., Chen, X., Zamanighomi, M., Zeng, W., Satpathy, A. T., Chang, H. Y., Wang, Y., and Wong, W. H. (2018). Integrative analysis of single-cell genomics data by coupled nonnegative matrix factorizations. Proc. Natl. Acad. Sci. , 115(30):7723-7728.

Ericsson, L., Gouk, H., Loy, C. C., and Hospedales, T. M. (2022). Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Process. Mag. , 39(3):42-62.

Eriksson, J. and Koivunen, V. (2004). Identifiability, separability, and uniqueness of linear ICA models. IEEE Signal Process. Lett. , 11(7):601-604.

Esposito, F., Scarabino, T., Hyvärinen, A., Himberg, J., Formisano, E., Comani, S., Tedeschi, G., Goebel, R., Seifritz, E., and Di Salle, F. (2005). Independent component analysis of fMRI group studies by self-organizing clustering. Neuroimage , 25(1):193-205.

Essen, D. C. V., Smith, S. M., Barch, D. M., Behrens, T. E., Yacoub, E., and Ugurbil, K. (2013). The WU-minn human connectome project: An overview. Neuroimage , 80:62-79.

Gentle, J. E. (1998). Numerical linear algebra for applications in statistics . Statistics and Computing. SpringerVerlag, New York.

Gessel, I. and Viennot, G. (1985). Binomial determinants, paths, and hook length formulae. Adv. in Math. , 58(3):300-321.

Gossi, F., Pati, P., Chouvardas, P., Martinelli, A. L., Kruithof-de Julio, M., and Rapsomaniki, M. A. (2023). Matching single cells across modalities with contrastive learning and optimal transport. Brief. Bioinform. , 24(3):bbad130.

Huang, B., Low, C. J. H., Xie, F., Glymour, C., and Zhang, K. (2022). Latent hierarchical causal structure discovery with rank constraints. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 5549-5561. Curran Associates, Inc.

Hyvärinen, A. (1999). Fast and robust fixed-point algorithms for independent component analysis. IEEE Trans. Neural Netw. , 10(3):626-634.

Hyvärinen, A. and Oja, E. (2000). Independent component analysis: algorithms and applications. Neural Netw. , 13(4):411-430.

Hyvärinen, A. and Ramkumar, P. (2013). Testing independent component patterns by inter-subject or intersession consistency. Front. Hum. Neurosci. , 7:94.

Khemakhem, I., Kingma, D., Monti, R., and Hyvärinen, A. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. In Chiappa, S. and Calandra, R., editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 2207-2217. PMLR.

Kim, T., Lee, I., and Lee, T.-W. (2006). Independent vector analysis: definition and algorithms. In 2006 Fortieth Asilomar Conference on Signals, Systems and Computers , pages 1393-1396. IEEE.

Klami, A., Virtanen, S., Leppäaho, E., and Kaski, S. (2014). Group factor analysis. IEEE Trans. Neural Netw. Learn. Syst. , 26(9):2136-2147.

Li, X.-L., Adalı, T., and Anderson, M. (2011). Joint blind source separation by generalized joint diagonalization of cumulant matrices. Signal Process. , 91(10):2314-2322.

Lin, Y., Wu, T.-Y., Wan, S., Yang, J. Y., Wong, W. H., and Wang, Y. (2022). scJoint integrates atlas-scale single-cell RNA-seq and ATAC-seq data with transfer learning. Nat. Biotechnol. , 40(5):703-710.

Lindström, B. (1973). On the vector representations of induced matroids. Bull. London Math. Soc. , 5:85-90.

Liu, J., Huang, Y., Singh, R., Vert, J.-P., and Noble, W. S. (2019). Jointly embedding multiple single-cell omics measurements. In Huber, K. T. and Gusfield, D., editors, 19th International Workshop on Algorithms in Bioinformatics (WABI 2019) , volume 143 of Leibniz International Proceedings in Informatics (LIPIcs) , pages 10:1-10:13, Dagstuhl, Germany. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

Liu, Y., Zhang, Z., Gong, D., Gong, M., Huang, B., Hengel, A. v. d., Zhang, K., and Shi, J. Q. (2022). Identifying weight-variant latent causal models. arXiv preprint arXiv:2208.14153 .

Lopez, R., Tagasovska, N., Ra, S., Cho, K., Pritchard, J., and Regev, A. (2022). Learning causal representations of single cells via sparse mechanism shift modeling. In NeurIPS 2022 Workshop on Causality for Real-world Impact .

Lukic, A. S., Wernick, M. N., Hansen, L. K., and Strother, S. C. (2002). An ICA algorithm for analyzing multiple data sets. In Proceedings. International Conference on Image Processing , volume 2, pages 821-824. IEEE.

Lyche, T. (2020). Numerical linear algebra and matrix factorizations , volume 22 of Texts in Computational Science and Engineering . Springer, Cham. With a foreword by Geir Dahl.

Maneshi, M., Vahdat, S., Gotman, J., and Grova, C. (2016). Validation of shared and specific independent component analysis (SSICA) for between-group comparisons in fMRI. Front. Neurosci. , 10:417.

McQuin, C., Goodman, A., Chernyshev, V., Kamentsky, L., Cimini, B. A., Karhohs, K. W., Doan, M., Ding, L., Rafelski, S. M., Thirstrup, D., Wiegraebe, W., Singh, S., Becker, T., Caicedo, J. C., and Carpenter, A. E. (2018). CellProfiler 3.0: Next-generation image processing for biology. PLOS Biol. , 16(7):e2005970.

Mesters, G. and Zwiernik, P. (2022). Non-independent components analysis. arXiv preprint arXiv:2206.13668 .

Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Xu, J., Bartsch, A. J., Jbabdi, S., Sotiropoulos, S. N., Andersson, J. L. R., Griffanti, L., Douaud, G., Okell, T. W., Weale, P., Dragonu, I., Garratt, S., Hudson, S., Collins, R., Jenkinson, M., Matthews, P. M., and Smith, S. M. (2016). Multimodal population brain imaging in the UK biobank prospective epidemiological study. Nat. Neurosci. , 19(11):1523-1536.

Nielsen, A. A. (2002). Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing data. IEEE Trans. Image Process. , 11(3):293-305.

Pandeva, T. and Forré, P. (2023). Multi-view independent component analysis with shared and individual sources. In Evans, R. J. and Shpitser, I., editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 1639-1650. PMLR.

Richard, H., Ablin, P., Thirion, B., Gramfort, A., and Hyvärinen, A. (2021). Shared independent component analysis for multi-subject neuroimaging. Adv. Neural Inf. Process. Syst. , 34:29962-29971.

Roeder, G., Metz, L., and Kingma, D. (2021). On linear identifiability of learned representations. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9030-9039. PMLR.

Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. (2021). Toward causal representation learning. Proc. IEEE , 109(5):612-634.

Shafto, M. A., Tyler, L. K., Dixon, M., Taylor, J. R., Rowe, J. B., Cusack, R., Calder, A. J., Marslen-Wilson, W. D., Duncan, J., Dalgleish, T., Henson, R. N., Brayne, C., and Matthews, F. E. (2014). The cambridge centre for ageing and neuroscience (cam-CAN) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing. BMC Neurol. , 14(1).

Shimizu, S., Inazumi, T., Sogawa, Y., Hyvärinen, A., Kawahara, Y., Washio, T., Hoyer, P. O., and Bollen, K. (2011). DirectLiNGAM: a direct method for learning a linear non-Gaussian structural equation model. J. Mach. Learn. Res. , 12:1225-1248.

Silva, R., Scheine, R., Glymour, C., and Spirtes, P. (2006). Learning the structure of linear latent variable models. J. Mach. Learn. Res. , 7(8):191-246.

Squires, C., Seigal, A., Bhate, S. S., and Uhler, C. (2023). Linear causal disentanglement via interventions. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 32540-32560. PMLR.

Stuart, T., Butler, A., Hoffman, P., Hafemeister, C., Papalexi, E., Mauck, W. M., Hao, Y ., Stoeckius, M., Smibert, P., and Satija, R. (2019). Comprehensive integration of single-cell data. Cell , 177(7):1888-1902.

Sui, J., Adali, T., Pearlson, G. D., and Calhoun, V. D. (2009). An ICA-based method for the identification of optimal FMRI features and components using combined group-discriminative techniques. Neuroimage , 46(1):73-86.

Svensén, M., Kruggel, F., and Benali, H. (2002). ICA of fMRI group study data. Neuroimage , 16(3):551-563.

van der Vaart, A. W. and Wellner, J. A. (1996). Weak convergence and empirical processes . Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics.

Varoquaux, G., Sadaghiani, S., Poline, J. B., and Thirion, B. (2009). CanICA: Model-based extraction of reproducible group-level ICA patterns from fMRI time series. In Medical Image Computing and Computer Aided Intervention , page 1.

Wang, X., Hutchinson, R., and Mitchell, T. M. (2003). Training fmri classifiers to detect cognitive states across multiple human subjects. In Thrun, S., Saul, L., and Schölkopf, B., editors, Proceedings of the 16th International Conference on Neural Information Processing Systems , volume 16, pages 709-716. MIT Press.

Wang, Y. S. and Drton, M. (2020). High-dimensional causal discovery under non-Gaussianity. Biometrika , 107(1):41-59.

Welch, J. D., Hartemink, A. J., and Prins, J. F. (2017). MATCHER: manifold alignment reveals correspondence between single cell transcriptome and epigenome dynamics. Genome Biol. , 18(1):1-19.

Xie, F., Cai, R., Huang, B., Glymour, C., Hao, Z., and Zhang, K. (2020). Generalized independent noise condition for estimating latent variable causal graphs. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Adv. in Neural Inf. Process. Syst. , volume 33, pages 14891-14902. Curran Associates, Inc.

Xie, F., Huang, B., Chen, Z., He, Y., Geng, Z., and Zhang, K. (2022). Identification of linear non-Gaussian latent hierarchical structure. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 24370-24387. PMLR.

Yang, K. D., Belyaeva, A., Venkatachalapathy, S., Damodaran, K., Katcoff, A., Radhakrishnan, A., Shivashankar, G. V., and Uhler, C. (2021a). Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nat. Comm. , 12(1).

Yang, K. D. and Uhler, C. (2019). Multi-domain translation by learning uncoupled autoencoders. Computational Biology Workshop, International Conference on Machine Learning .

Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., and Wang, J. (2021b). Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9593-9602.

Yuan, Q. and Duren, Z. (2022). Integration of single-cell multi-omics data by regression analysis on unpaired observations. Genome Biol. , 23(1):1-19.

Zeng, Y., Shimizu, S., Cai, R., Xie, F., Yamamoto, M., and Hao, Z. (2021). Causal discovery with multi-domain lingam for latent factors. In Zhou, Z.-H., editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2097-2103. International Joint Conferences on Artificial Intelligence Organization. Main Track.

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 2242-2251.

Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. (2021). A comprehensive survey on transfer learning. Proc. IEEE , 109(1):43-76.

A Proofs

Proof of Theorem 3.1. Let P X ∈ M ( G m ) for an m -domain graph G m = ( H∪ V, D ) with shared latent nodes L = [ ℓ ] and representation ( B,P ) . By Condition (C1) the measure P has independent, non-degenerate, non-symmetric marginals P i , i ∈ H with mean zero and variance one. Moreover, since B ∈ Im ( ϕ G m ) , we have B = G ( I -A ) -1 for matrices G ∈ R D V H and A ∈ R D HH .

Fix one domain e ∈ [ m ] . Recall that we denote by S e = pa ( V e ) = L∪ I e the set of latent parents in domain e . Define the matrix

and observe that we can write P X e = B V e , H # P = B e # P S e . This is due to the fact that G V e , H\ S e = 0 and [( I -A ) -1 ] S e , H\ S e = 0 by the definition of an m -domain graph.

In particular, the equality P X e = B e # P S e shows that the representation in Line 4 of Algorithm 1 exists. Now, we show that it is unique up to signed permutation by applying results on identifiability of linear ICA. Since G e has full column rank by Condition (C2) and [( I -A ) -1 ] S e ,S e is invertible, the matrix B e also has full column rank. Let P X e = ̂ B e # P e be any representation, where ̂ B e ∈ R d e × ̂ s e and P e is an ̂ s e -dimensional probability measure with independent, non-degenerate marginals P e i . Due to Condition (C1), all probability measures P i are non-Gaussian and non-degenerate and therefore we have by Eriksson and Koivunen (2004, Theorem 3 and 4) the identities

where Λ e is an s e × s e diagonal matrix with nonzero entries and R e is an s e × s e permutation matrix. In particular, we have ̂ s e = s e , which means that ̂ B e ∈ R d e × s e and that P e is an s e -dimensional probability measure. Line 4 also requires that each marginal P e i has unit variance. This removes the scaling indeterminacy in (3) and we have

where D e is a diagonal matrix with entries D e ii ∈ {± 1 } . In particular, this means that the distributions P e and P S e coincide up to permutation and sign of the marginals.

The matching in Line 6 identifies which components of P e are shared. By Condition (C1), two components of different domains P e i and P f j are shared if and only if they coincide up to sign, that is, if and only if d ( P e i , P f j ) = 0 or d ( P e i , -P f j ) = 0 . If their distribution coincide up to sign, than either d ( P e i , P f j ) = 0 or d ( P e i , -P f j ) = 0 but not both since Condition (C1) requires the distribution of the error variables to be non-symmetric. We conclude that in each domain e ∈ [ m ] there exists an s e × s e signed permutation matrix Q e such that

̸

for all i = 1 , . . . , ̂ ℓ and for all f = e . In particular, ̂ ℓ = ℓ and ̂ L = L .

It remains to show that ̂ B = B Ψ and ̂ P = Ψ ⊤ # P for a signed permutation block matrix Ψ ∈ Π . By Equation (4), the distributions [( Q e ) ⊤ # P e ] L and [( Q e ) ⊤ # P e ] L coincide, which means that

where Ψ L is an ℓ × ℓ signed permutation matrix and Ψ I e is an | I e | × | I e | signed permutation matrix. Importantly, the matrix Ψ ⊤ L does not depend on the domain e ∈ [ m ] . Hence, the matrix Φ e := R e D e Q e is a signed permutation matrix with block structure as in Equation (5). Moreover, we have

which means that the matrix ̂ B can be factorized as

where Ψ ∈ Π . Similarly, we have for all e ∈ [ m ] ,

We conclude that

Before proving Lemma 4.3 and Theorem 4.4 we fix some notation. Let G m = ( V ∪ H , D ) be an m -domain graph. We denote by anc ( v ) = { k ∈ H : there is a directed path k → ··· → v in G m } the ancestors of a node v ∈ V . For subsets W ⊆ V , we denote anc ( W ) = ⋃ v ∈ W anc ( w ) . Moreover, for L ⊆ H and v ∈ V , we write shortly pa L ( v ) = pa ( v ) ∩ L .

Proof of Lemma 4.3. Let B ∈ Im ( ϕ G m ) . Then we can write B = G · ( I -A ) -1 with

Moreover, observe that, by the definition of an m -domain-graph, the matrix B V, L factorizes as

Now, suppose that i and j are partial pure children of a fixed node k ∈ L . Then pa L ( i ) = { k } = pa L ( j ) . In particular, the only entry that may be nonzero in the row G i, L is given by G ik and the only entry that may be nonzero in the row G j, L is given by G jk . Thus, we have

̸

̸

Similarly, it follows that B j, L = G jk [( I -A ) -1 ] k, L . This means that the row B j, L is a multiple of the row B i, L and we conclude that rank ( B { i,j } , L ) ≤ 1 . Equality holds due to the faithfulness condition (C4) which implies that B ik = 0 and B jk = 0 , i.e., B { i,j } , L is not the null matrix.

For the other direction suppose that rank ( B { i,j } , L ) = 1 . By applying the Lindström-Gessel-Viennot Lemma (Gessel and Viennot, 1985; Lindström, 1973) equivalently as in Dai et al. (2022, Theorem 1 and 2), it can be seen that

where S is a vertex cut from anc ( L ) to { i, j } if and only if there exists no directed path in G m from anc ( L ) to { i, j } without passing through S . Moreover, equality holds in (6) for generic (almost all) choices of parameters. Since we assumed rank faithfulness in Condition (C4) we exclude cases where the inequality is strict and therefore have equality. By the definition of an m -domain graph we have that anc ( L ) = L . Thus, if rank ( B { i,j } , L ) = 1 , there must be a single node k ∈ L such that { k } is a vertex cut from L to { i, j } . But then it follows that i and j have to be partial pure children of k by the definition of an m -domain graph and by using the assumption that there are no zero-rows in B L .

To prove Theorem 4.4 we need the following auxiliary lemma.

̸

Lemma A.1. Let G = ( V, D ) be a DAG with topologically ordered nodes V = [ p ] and let M be a lower triangular matrix with entries M ii = 0 for all i = 1 , . . . , p and M ij = 0 if and only if there is a directed path j →··· → i in G . Let Q σ 1 and Q σ 2 be permutation matrices. Then the matrix Q σ 1 MQ σ 2 is lower triangular if and only if σ 2 = σ -1 1 and σ 2 ∈ S ( G ) .

Proof of Lemma A.1. By the definition of a permutation matrix, we have

First, suppose that σ 2 = σ -1 1 and σ 2 ∈ S ( G ) , and let i, j ∈ [ p ] such that σ 2 ( i ) < σ 2 ( j ) . Then, by the definition of S ( G ) , there is no directed path j →··· → i in the graph G and therefore we have M ij = 0 . But this means that [ Q σ 1 MQ σ 2 ] σ 2 ( i ) σ 2 ( j ) = 0 and we conclude that the matrix Q σ 1 MQ σ 2 is lower triangular.

̸

Now, assume that Q σ 1 MQ σ 2 is lower triangular, where σ 1 and σ 2 are arbitrary permutations on the set [ p ] . Since M has no zeros on the diagonal, we have M ii = [ Q σ 1 MQ σ 2 ] σ -1 1 ( i ) σ 2 ( i ) = 0 for all i = 1 , . . . , p . It follows that σ -1 1 ( i ) ≥ σ 2 ( i ) for all i = 1 , . . . , p because Q σ 1 MQ σ 2 is lower triangular. But this is only possible if the permutations coincide on all elements, i.e., we have σ 2 = σ -1 1 . It remains to show that σ 2 = σ -1 1 ∈ S ( G ) . For any edge j → i ∈ D we have that M ij = 0 . Recalling Equation (7) this means that [ Q σ 1 MQ σ 2 ] σ 2 ( i ) σ 2 ( j ) = 0 . But since Q σ 1 MQ σ 2 is lower triangular this can only be the case if σ 2 ( j ) < σ 2 ( i ) which proves that σ 2 ∈ S ( G ) .

̸

̸

Proof of Theorem 4.4. Each latent node in L has two partial pure children by Condition (C3). After removing zero-rows in Line 3 of Algorithm 2 it holds by Lemma 4.3 that rank ( B ∗ { i,j } , L ) = 1 if and only if there is a latent node in L such that i and j are both partial pure children of that latent node. Hence, each tuple ( i k , j k ) k ∈L in Line 4 of Algorithm 2 consists of two partial pure children of a certain latent node. The requirement rank ( B ∗ { i k ,i q } , L ) = 2 ensures that each pair of partial pure children has a different parent.

By the definition of an m -domain-graph and the fact that B ∗ = ̂ B L , for I = { i 1 , . . . , i ℓ } , we have the factorization

where G ∈ R D V H , A ∈ R D HH and Ψ L is a signed permutation matrix. Let Q 1 and Q 2 be permutation matrices and let Λ be a diagonal matrix with non-zero diagonal elements and let D be a diagonal matrix with entries in {± 1 } . Then we can rewrite Equation (8) as

Now, we apply Lemma A.1. Since we assume throughout Section 4 that the nodes L are topologically ordered, the matrix M is lower triangular with no zeros on the diagonal. Moreover, by Condition (C4) we have M ij = 0 if and only if there is a directed path j →··· → i in G L . In Line 6 we find other permutation matrices R 1 and R 2 such that

̸

̸

is lower triangular. Now, define the permutation matrices Q σ 1 = R 1 Q 1 and Q σ 2 = Q 2 R 2 . Then we have by Lemma A.1 that Q σ 1 = Q ⊤ σ 2 and that σ 2 ∈ S ( G L ) . Hence, the matrix W factorizes as

where ˜ Λ and ˜ D are diagonal matrices with the entries given by permutations of the entries of Λ and D . Lines 7 and 8 address the scaling and sign matrices ˜ Λ and ˜ D . In particular, we have that ˜ W ′ = D ′ Q ⊤ σ 2 ( I -A L , L ) -1 Q σ 2 D ′ for another diagonal matrix D ′ with entries in {± 1 } , since each entry on the diagonal of ˜ W ′ is equal to 1 . Thus, we have

Since Q σ 2 D ′ is a signed permutation matrix with σ 2 ∈ S ( G L ) , the first part of the theorem is proved. If G vk > 0 whenever v is a pure child of k , the matrix ˜ Λ only has positive entries which means that D ′ is equal to the identity matrix. This proves the second part.

B Additional Examples

The graph in Figure 4 is an m -domain graph corresponding to the compact version in Figure 2 in the main paper.

Example B.1. Consider the m -domain graph in Figure 4. The linear structural equation model among the latent variables is determined by lower triangular matrices of the form

Moreover, the domain-specific mixing matrices G e are of the form

Since the shared latent nodes are given by L = { 1 , 2 } , we have

and

3

4

1

2

5

v

1

1

v

1

2

v

1

3

v

1

4

v

2

1

v

2

2

v

2

3

v

2

4

v

2

5

Example B.2. Consider the m -domain graph in Figure 4. The partial pure children of node 1 ∈ L are given by { v 1 2 , v 2 1 } and the partial pure children of 2 ∈ L are given by { v 1 3 , v 2 3 } . Moreover, by continuing Example B.1, we have that the matrix B L is given by

It is easy to see that the two submatrices

have rank one. The first matrix corresponds to the partial pure children { v 1 2 , v 2 1 } in the graph in Figure 4 while the second matrix correspond to the partial pure children { v 1 3 , v 2 3 } . Note that the rank of any other 2 × 2 submatrix is generically (i.e., almost surely) equal to 2 .

C Discussion of the Assumptions

In this section, we discuss aspects of Conditions (C1)-(C3) that allow for identifiability. In particular, we discuss the necessity of pairwise different and non-Gaussian error distributions if one is not willing to make further assumptions. Moreover, we elaborate on the sparsity conditions on the mixing matrix and explain why some sparsity assumption is necessary.

Pairwise Different Error Distributions. Given any two potentially different m -domain graphs G m = ( H∪ V, D ) and ˜ G m = ( ˜ H∪ V, ˜ D ) , identifiability of the joint distribution in multi-domain causal representation models means that

for any representation ( B,P ) of a distribution in M ( G m ) and any representation ( ˜ B, ˜ P ) of a distribution in M ( ˜ G m ) , where the matrices B V e ,S e and ˜ B V e , ˜ S e have full column rank for all e ∈ [ m ] . The left-hand side says that the marginal distributions in each domain are equal, while the right-hand side says that the joint distributions are equal. If there are m -domain graphs, such that the left-hand sides holds but the right-hand side is violated, then we say that the joint distribution is not identifiable .

We assume in this section that the marginal error distributions P i , i ∈ H are non-Gaussian and have unit variance, but are not necessarily pairwise different or non-symmetric. Then the right-hand side holds if and only if the number of shared latent nodes in each graph is equal, i.e., ℓ = ˜ ℓ , and there is a signed permutation matrix Ψ such that B = ˜ B Ψ and P = Ψ ⊤ # ˜ P . Here, the matrix Ψ does

̸

not necessarily have a block structure. The equivalence is implied by the identifiability of the usual, one-domain linear ICA ( see, e.g., Buchholz et al. (2022)) together with the fact that for ℓ = ˜ ℓ , we have |H| ̸ = | ˜ H| and, therefore, the distributions on the right-hand have support over different dimensional subspaces.

Theorem 3.1 shows that assumptions (C1) and (C2) are sufficient for identifiability of the joint distribution. In particular, we show that they imply identifiability in a stronger sense, namely, that it follows from the left-hand side that ℓ = ˜ ℓ and B = ˜ B Ψ and P = Ψ ⊤ # ˜ P for a signed permutation Ψ ∈ Π with block structure. The next proposition reveals necessary conditions for identifiability.

Proposition C.1. Let G m be an m -domain graph with shared latent nodes L = [ ℓ ] , and let P X ∈ M ( G m ) with representation ( B,P ) . Suppose that m ≥ 2 and that everything except the assumption about pairwise different error distributions in Conditions (C1) and (C2) is satisfied. Then, the joint distribution is not identifiable if one of the following holds:

(i) There is i, j ∈ L such that P i = P j or P i = -P j .

(ii) There is i ∈ L and j ∈ I e for some e ∈ [ m ] such that P i = P j or P i = -P j .

̸

(iii) For all e ∈ [ m ] there is i e ∈ I e such that P i e = P j f or P i e = -P i f for all e = f .

Proof. For each of the three cases, we will construct another m -domain graph G m = ( H∪ V, D ) such that for suitable representations ( ˜ B, ˜ P ) of distributions in M ( ˜ G m ) , the left-hand side of (9) holds, but the right-hand side is violated.

To prove the statement for case (i), let i, j ∈ L and assume that P i = P j . We define the m -domain graph ˜ G m = ( ˜ H∪ V, ˜ D ) to be the almost same graph as G m = ( H∪ V, D ) , we only 'swap' the roles of the latent nodes i and j on an arbitrary domain e ∈ [ m ] . That is, for each v ∈ V e , if there was an edge i → v in D , we remove that edge from ˜ D and add the edge j → v instead, and vice versa. Otherwise, the graph ˜ G m has the same structure as G m . Now, let ˜ P = P and define a the matrix ˜ B to be the same matrix as B , except for the subcolumns ˜ B V e ,i := B V e ,j and ˜ B V e ,j := B V e ,i , that is, we swapped B V e ,i and B V e ,j . Then the pair ( ˜ B, ˜ P ) is a representation of some distribution in M ( ˜ G m ) . Recall from the proof of Theorem 3.1 that Condition (C2) implies that the matrix B V e ,S e has full column rank. Since we only swapped columns in ˜ B V e , ˜ S e , it still has full column rank. Moreover, observe that the left hand side of (9) is satisfied since P i = P j , that is, the marginal distributions on the single domains coincide.

However, now consider another domain f ∈ [ m ] and the submatrices

Since all of the four subcolumns are nonzero and neither B V e ,j is equal to B V e ,i nor B V f ,j is equal to B V f ,i , there is no signed permutation matrix Ω such that B V e ∪ V f , { i,j } = ˜ B V e ∪ V f , { i,j } Ω . Hence, there is also no larger signed permutation matrix Ψ such that B = ˜ B Ψ . We conclude that the right-hand side of (9) is violated and the joint distribution is not identifiable. Finally, note that the above arguments also hold if P i = -P j by adding ' -' signs in appropriate places.

The proof for case (ii) works with exactly the same construction. That is, for i ∈ L and j ∈ I e we swap the roles of i and j on the domain e . Then, for any other domain f ∈ [ m ] , we obtain the submatrices

By the same arguments as before, this shows that there is no signed permutation matrix Ψ such that B = ˜ B Ψ and, hence, the joint distribution is not identifiable.

̸

To prove case (iii), we consider a slightly different construction. However, we also assume that P i e = P i f for all e = f , since for P i e = -P i f we only have to add some ' -' signs in the following. We define the m -domain graph ˜ G m = ( ˜ H∪ V, ˜ D ) by identifying the nodes i e , e ∈ [ m ] with a new

node k . That is, ˜ L = L ∪ { k } and ˜ H = ( ⋃ e ∈ [ m ] I e \ { i e } ) ∪ ˜ L . For i ∈ ˜ H\{ k } and v ∈ V , the edge set ˜ D contains an edge i → v if and only if the edge i → v is in D . For the node k ∈ ˜ H and v ∈ V , we put an edge k → v in ˜ D if and only if there is an edge i e → v in D for some e ∈ [ m ] .

Now, define the matrix ˜ B such that ˜ B V, ˜ H\{ k } := B V, ˜ H\{ k } and ˜ B V e ,k := B V e ,i e for all e ∈ [ m ] . Then the pair ( ˜ B, ˜ P ) is a representation of some distribution in M ( ˜ G m ) . Moreover, each submatrix ˜ B V e , ˜ S e is equal to B V e ,S e up to relabeling of the columns. That is, the column that is labeled by i e in B V e ,S e is now labeled by k in ˜ B V e , ˜ S e . We define the measure ˜ P such that ˜ P ˜ H\{ k } = P ˜ H\{ k } and ˜ P k = P i e for all e ∈ [ m ] . Then the pair ( ˜ B, ˜ P ) is a representation of some distribution in M ( ˜ G m ) and, in particular, the left hand side of (9) is satisfied. That is, the marginal distributions coincide on each domain. However, the number of shared latent variables in both m -domain graphs is different since we have ˜ ℓ = ℓ +1 . We conclude that the joint distribution is not identifiable.

The proposition states that it is in most cases necessary that error distributions are pairwise different. However, in two cases the same error distributions still lead to identifiability. First, if i, j ∈ I e , then the corresponding error distributions may be the same and the joint distribution is still identifiable. Similarly, if there are latent nodes i e in a few domains e ∈ [ m ] such that the corresponding error distributions coincide, but there is at least one domain f ∈ [ m ] where there is no latent node with the same error distribution, then the joint distribution is also identifiable. Both can be seen by taking the the proofs of Theorem 3.1 and Proposition C.1 together.

Gaussian Errors. Without additional assumptions to those in Section 3, it is impossible to recover the joint distribution if the distributions of the errors ε i of the latent structural equation model in Equation (1) are Gaussian. In this case, the distribution of Z as well as the distribution of each observed random vector X e is determined by the covariance matrix only. The observed covariance matrix in domain e ∈ [ m ] is given by Σ e = G e Cov [ Z L∪ I e ]( G e ) ⊤ . However, knowing Σ e gives no information about Z L∪ I e other than rank (Σ e ) = |L| + | I e | , that is, we cannot distinguish which latent variables are shared and which ones are domain-specific. This is formalized in the following lemma.

Lemma C.2. Let Σ be any d × d symmetric positive semidefinite matrix of rank p and let Ξ be another arbitrary p × p symmetric positive definite matrix. Then there is G ∈ R d × p such that Σ = G Ξ G ⊤ .

Proof. Let Σ be a d × d symmetric positive semidefinite matrix of rank p . Then, Σ has a decomposition similar to the Cholesky decomposition; see e.g. Gentle (1998, Section 3.2.2). That is, there exists a unique matrix T , such that A = TT ⊤ , where T is a lower triangular matrix with p positive diagonal elements and d -p columns containing all zeros. Define ˜ T to be the d × p matrix containing only the non-zero columns of T .

On the other hand, let Ξ be a symmetric positive definite p × p matrix. By the usual Cholesky decomposition (Lyche, 2020, Section 4.2.1), there exists a unique p × p lower triangular matrix L with positive diagonal elements such that Ξ = LL ⊤ . Now, define G := ˜ TL -1 ∈ R d × p . Then,

Due to Lemma C.2 it is necessary to consider non-Gaussian distributions to obtain identifiability of the joint distribution.

Example C.3. In the Gaussian case we cannot distinguish whether the two observed domains in Figure 5 share a latent variable or not. Said differently, the observed marginal distributions may either be generated by the mechanism defined by graph (a) or graph (b) and there is no way to distinguish from observational distributions only.

Sparsity Assumptions. Let B ∈ Im ( ϕ G m ) for an m -domain graph G m = ( H∪ V, D ) and suppose that we are given the matrix B L = G V, L ( I -A L , L ) -1 , that is, we are given the submatrix with columns indexed by the shared latent nodes. Now, assume that the graph does not impose any sparsity restrictions on G V, L , which means that the set R D V L of possible matrices G V, L is equal to R | V |×|L| . Then, the set of possible matrices B L is also unrestricted, that is, B L can be any matrix in R | V |×|L| no matter the form of the matrix A L , L ∈ R D LL . In other words, for arbitrary shared latent graphs

1

2

3

V

1

V

2

(a)

1

2

3

4

V

1

V

2

(b)

G L = ( L , D LL ) and arbitrary corresponding parameter matrices A L , L ∈ R D LL , we don't get any restrictions on the matrix B L . Therefore, it is impossible to infer A L , L from B L .

Condition (C3) requires that there are two partial pure children for every shared latent node k ∈ L , which implies that there are 2 |L| rows in G V, L in which only one entry may be nonzero. While we show in Theorem 4.4 that this condition is sufficient for identifiability of A L , L , we leave it open for future work to find a necessary condition.

D Algorithms for Finite Samples

We adjust Algorithm 1 such that it is applicable in the empirical data setting. That is, rather than the exact distribution P X e , we have a matrix of observations X e ∈ R d e × n e in each domain e ∈ [ m ] . The sample size n e might be different across domains. We denote n min = min e ∈ [ m ] n e and n max = max e ∈ [ m ] n e . For implementing linear ICA on finite samples, multiple well developed algorithms are available, e.g., FastICA (Hyvärinen, 1999; Hyvärinen and Oja, 2000), Kernel ICA (Bach and Jordan, 2003) or JADE (Cardoso and Souloumiac, 1993). Applying them, we obtain a measure ̂ P e i which is an estimator of the true measure P e i in Algorithm 1, Line 4.

The remaining challenge is the matching in Line 6 of Algorithm 1. For finite samples, the distance between empirical distributions is almost surely not zero although the true underlying distributions might be equal. In this section, we provide a matching strategy based on the two-sample KolmogorovSmirnov test (van der Vaart and Wellner, 1996, Section 3.7). We match two distributions if they are not significantly different. During this process, there might occur false discoveries, that is, distributions are matched that are actually not the same. We show that the probability of falsely discovering shared nodes shrinks exponentially with the number of domains.

For two univariate Borel probability measures P i , P j , with corresponding cumulative distribution functions F i , F j , the Kolmogorov-Smirnov distance is given by the L ∞ -distance

The two-sample Kolmogorov-Smirnov test statistic for the null hypothesis H 0 : d KS ( P e i , P f j ) = 0 is given by

It is important to note that ̂ P e i is not an empirical measure in the classical sense since it is not obtained from data sampled directly from the true distribution P e i . In addition to the sampling error there is the uncertainty of the ICA algorithm. However, in the analysis we present here, we will neglect this error and treat ̂ P e i as an empirical measure. In this case, under H 0 , the test statistic T ( ̂ P e i , ̂ P f j ) converges in distribution to ∥ B ∥ ∞ , where B is a Brownian bridge from 0 to 1 (van der Vaart and Wellner, 1996, Section 2.1). For a given level α ∈ (0 , 1) , we choose the critical value as c α = inf { t : P ( ∥ B ∥ ∞ > t ) ≤ α } and reject H 0 if T ( ̂ P e i , ̂ P f j ) > c α .

Definition D.1. Let α ∈ (0 , 1) and suppose the distributions { ̂ P e 1 , . . . , ̂ P e ̂ s e } and { ̂ P f 1 , . . . , ̂ P f ̂ s f } are given for two domains e, f ∈ [ m ] . Define

Definition D.1 essentially states that two measures are matched if the test statistic (10) is not significantly large and the null hypothesis cannot be rejected. Taking the minimum of T ( ̂ P e i , ̂ P f j ) and T ( ̂ P e i , -̂ P f j ) accounts for the sign indeterminacy of linear ICA. For two fixed domains e, f ∈ [ m ] , if it happens that the statistic T ef ij for multiple pairs ( i, j ) is small enough, then the pair with the minimal value of the statistic is matched. Note that one may use any other test than the Kolmogorov-Smirnov test to define a matching as in Definition D.1. We discover a shared latent node if it is matched consistently across domains.

Definition D.2. Let C = ( i 1 , . . . , i m ) be a tuple with m elements such that i e ∈ [ ̂ s e ] . Then we say that C determines a shared node if Ω α ( ̂ P e i e , ̂ P f i f ) = 1 for all i e , i f ∈ C .

Inferring the existence of a shared node which does not actually exist may be considered a more serious error than inferring a shared node determined by a set C , where only some components of C are wrongly matched. In the following theorem we show that the probability of falsely discovering shared nodes shrinks exponentially with the number of wrongly matched components.

Theorem D.3. Let C = ( i 1 , . . . , i m ) be a tuple with m elements such that i e ∈ [ ̂ s e ] . Let g : N × R ≥ 0 → R ≥ 0 be a function that is monotonically decreasing in n ∈ N and assume the following:

̸

(ii) There is E ⊆ [ m ] with | E | ≥ 2 and a constant κ > 0 such that d KS ( P e i e , P f i f ) ≥ κ and d KS ( P e i e , -P f i f ) ≥ κ for all e ∈ E , f ∈ [ m ] with e = f .

Then

Proof of Theorem D.3. Let C = ( i 1 , . . . , i m ) and g be as in the statement of the theorem. W.l.o.g. we assume that E = { 1 , . . . , | E |} and that T ( ̂ P e i e , ̂ P f i f ) ≤ T ( ̂ P e i e , -̂ P f i f ) . Observe that C determines a shared node if and only if

Now, we have

By the triangle inequality we have

Moreover, if e ∈ E and f ∈ [ m ] , then we have by Condition (ii)

Using (12) and (13) together with the fact √ n e n f n e + n f ≥ n min √ 2 n max , we obtain the following chain of implications for e ∈ E and f ∈ [ m ] :

Now, consider the event ⋂ e ∈ E,f ∈ [ m ] e<f { T ( ̂ P e i e , ̂ P f i f ) ≤ c α } . On this event, there cannot be two elements e, f ∈ E such that both

To see this recall that E ⊆ [ m ] . We conclude that it must hold { d KS ( ̂ P e i e , P e i e ) ≥ κ 2 -√ n max √ 2 n min c α } for all but at most one element of E . We denote this exceptional element by e ∗ ∈ E . Taking up (11), we get the following:

The last three steps need more explanation: Equality (14) follows from the fact that domains are unpaired. That is, the distances d KS ( ̂ P e i e , P e i e ) and d KS ( ̂ P f i f , P f i f ) are pairwise independent for different domains e, f ∈ [ m ] . Equality (15) is trivial since d KS ( ̂ P e i e , P e i e ) ≥ 0 . Finally, Inequality (16) follows from Condition (i) and that the function g is monotonically decreasing in n . We also used the fact that | E \ { e ∗ }| = | E | -1 .

If ̂ P e i were an empirical measure in the classical sense, then Condition (i) in Theorem D.3 translates to the well-known Dvoretzky-Kiefer-Wolfowitz inequality, that is, the function g is given by g ( n, x ) =

Algorithm 3 IdentifyJointDistributionEmpirical

1: Hyperparameters: γ > 0 , α ∈ (0 , 1) .

2: Input: Matrix of observations X e ∈ R d e × n e for all e ∈ [ m ] .

3: Output: Number of shared latent variables ̂ ℓ , matrix ̂ B and probability measure ̂ P .

4: for e ∈ [ m ] do

5: Linear ICA: Use any linear ICA algorithm to obtain a mixing matrix ˜ B e ∈ R d e × ̂ s e , where ̂ s e = rank γ ( X e ( X e ) ⊤ ) . Compute the matrix ˜ η e = ( ˜ B e ) † X e ∈ R ̂ s e × n e , where ( ˜ B e ) † is the Moore-Penrose pseudoinverse of ˜ B e .

6: Scaling: Let ∆ e be a ̂ s e × ̂ s e diagonal matrix with entries ∆ e ii = 1 n e [ ˜ η e ( ˜ η e ) ⊤ ] ii . Define ̂ B e = ˜ B e (∆ e ) -1 / 2 and η e = (∆ e ) -1 / 2 ˜ η e .

7: Let ̂ P e be the estimated probability measure with independent marginals such that ̂ P e i is the empirical measure of the row η e i, ∗ .

8: end for

9: Matching: Let ̂ ℓ be the maximal number such that there is a signed permutation matrix Q e in each domain e ∈ [ m ] such that

̸

for all i = 1 , . . . , ̂ ℓ and all f = e , where α t = α/t with t = 2 ∑ e<f ̂ s e ̂ s f . Let ̂ L = { 1 , . . . , ̂ ℓ } .

10: Construct the matrix ̂ B and the tuple of probability measures ̂ P given by

11: return ( ̂ ℓ , ̂ B , ̂ P ).

Algorithm 4 IdentifySharedGraphEmpirical

1: Hyperparameters: γ > 0 .

2: Input: Matrix B ∗ ∈ R | V |× ℓ .

3: Output: Parameter matrix ̂ A ∈ R ℓ × ℓ .

4: Remove rows B ∗ i, L with ∥ B ∗ i, L ∥ 2 ≤ γ from the matrix B ∗ .

5: Find tuples ( i k , j k ) k ∈L with the smallest possible scores σ min ( B ∗ { i k ,j k } , L ) such that

̸

(i) i k = j k for all k ∈ L and { i k , j k } ∩ { i q , j q } = ∅ for all k, q ∈ L such that k = q and (ii) σ min ( B ∗ { i k ,i q } , L ) | > γ for all k, q ∈ L such that k = q .

6: Let I = { i 1 , . . . , i ℓ } and consider the matrix B ∗ I, L ∈ R ℓ × ℓ .

̸

7: Find two permutation matrices R 1 and R 2 such that W = R 1 B ∗ I, L R 2 is as close as possible to lower triangular. This can be measured, for example, by using ∑ i<j W 2 ij .

8: Multiply each column of W by the sign of its corresponding diagonal element. This yields a new matrix ˜ W with all diagonal elements positive.

9: Divide each row of ˜ W by its corresponding diagonal element. This yields a new matrix ˜ W ′ with all diagonal elements equal to one.

10: Compute ̂ A = I -( ˜ W ′ ) -1 .

11: return ̂ A .

̸

.

̸

2 exp( -2 nx 2 ) . Given a tuple C = ( i 1 , . . . , i m ) that defines a shared node, Condition (ii) is an assumption on the number of wrongly matched components. The most extreme case is when the shared node does not actually exist and all components are wrongly matched. That is, the measures ̂ P e i e and ̂ P f i f are matched even though d KS ( P e i e , P f i f ) = 0 and d KS ( P e i e , -P f i f ) = 0 for all e, f ∈ [ m ] . On the other hand, if | E |≪ m , then C determines a shared node where the majority of the components are correctly matched.

̸

If g ( n, x ) → 0 for n → ∞ and x > 0 , the statement of the theorem becomes meaningful under the constraint √ n max /n min → 0 . In this case, the probability that a given tuple C with wrong components E determines a shared node goes to zero for large sample sizes n min . As noted, the probability of falsely discovering a shared node decreases exponentially with the number of wrongly matched components | E | . In the extreme case, this means that the probability of falsely discovering shared nodes with all components wrongly matched, i.e., E = [ m ] , decreases exponentially with the number of domains m .

Theorem D.3 also tells us that the probability of falsely matching two measures ̂ P e i and ̂ P f j becomes zero if the sample size grows to infinity and the linear ICA algorithm is consistent. However, with finite samples we might fail to match two measures where the underlying true measures are actually the same, i.e., we falsely reject the true null hypothesis H 0 . Thus, we might be overly conservative in detecting shared nodes due to a high family-wise error rate caused by multiple testing. We suggest to correct the level α to account for the amount of tests carried out. One possibility is to apply a Bonferroni-type correction. The total number of tests is given by t = 2 ∑ e<f ̂ s e ̂ s f . This means that an adjusted level is given by α t = α/t and instead of the critical value c α we consider the adjusted critical value c α t .

Algorithm 3 is the finite sample version of Algorithm 1 with the matching Ω α defined in Definition D.1. To determine the number of independent components for the linear ICA step in each domain, we need to check the rank ( X e ( X e ) ⊤ ) . We specify the rank of a matrix M as number of singular values which are larger than a certain threshold γ and denote it by rank γ ( M ) .

In Algorithm 4 we also provide a finite sample version of Algorithm 2 where we only have the approximation B ⋆ = ̂ B L ≈ B L Ψ L for a signed permutation matrix Ψ L . For a matrix M , we denote by σ min ( M ) the smallest singular value.

E Error Distributions in Simulations

We specify L = { 1 , 2 , 3 } , I 1 = { 4 , 5 } , I 2 = { 6 , 7 } and I 3 = { 8 , 9 } . Note that the set I 3 does not exist if the number of domains is m = 2 . The error distributions in all simulations are specified as follows if not stated otherwise.

where the overline means that each distribution is standardized to have mean 0 and variance 1 . Figure 6 shows histograms of the empirical distributions.

F Additional Simulation Results

In this section, we make additional experiments. First, we consider another setup where all our assumptions are satisfied but we have more domains and more shared latent variables. Then, we also consider two setups where some of our assumptions are not satisfied.

Different Setup. We make additional experiments on a similar scale as in Section 5, but with more shared nodes and less domain specific nodes. This time, we consider ℓ = 5 shared latent nodes and | I e | = 1 domain-specific latent node in each domain. Moreover, we also consider m = 4 domains. The dimensions are given by d e = d/m for all e ∈ [ m ] and d = 48 . The graphs and edge weights

0.4

0.4

0.5

0.4

1.0

0.3

E'0

0.4

0.3

0.8

0.2

0.2

0.3

0.2

0.2

0.6

0.4

0.1

0.1

0.1

0.1

0.2

0.0

2

0

2

0.0

2

0

2

4

0.0

0.0

2.5

5.0

7.5

0.0

2.5

0.0

2.5

5.0

7.5

0.0

0

10

20

30

40

0.4

0.5

0.8

0.4

0.4

0.3

0.6

0.3

0.3

0.2

0.4

0.2

0.2

0.1

0.2

0.1

0.1

0.0

0.0

0.0

0.0

2

0

2

4

0.0

2.5

5.0

7.5

2

0

2

4

2

0

2

4

6

(a)

(b)

(c)

5

0.030

2domains

2.5

2domains

<

Average of

4

3

coreB

0

S

0.025

0.020

3domains

4domains

Median scoreA

2.0

1.5

3domains

4domains

ian

0.015

2

2domains

Medi

0.010

1.0

1

3domains

4domains

M

0.005

0.5

0

0.0

10

3

10°

4

10

3

10°

4

10

3

10°

4

Sample size

Sample size

Samplesize

are sampled equivalently as in Section 5 in the main paper. We also consider the same distribution of the error variables as specified in Appendix E, where we specify L = { 1 , 2 , 3 , 4 , 5 } , I 1 = { 6 } , I 2 = { 7 } , I 3 = { 8 } and I 4 = { 9 } .

Figure 7 shows the results where the scores are equivalent as in the main paper. Once again, we see that the estimation error for the matrices B L and A L , L decreases with increasing sample size. This supports our proof of concept and shows that the adapted algorithms are consistent for recovering B L and A L , L from finite samples.

Violated Assumptions. We consider the same setup as in the main paper in Section 5 with l = 3 shared latent nodes, but we fix the number of domains to m = 3 . In this experiment, we compare the results where data was generated such that all our assumptions are satisfied with two setups where we violate some of the assumptions. In the first setup, we violate Condition (C1) that requires pairwise different error distributions. We specify the error distributions as follows.

where, as before, the overline means that each distribution is standardized to have mean 0 and variance 1 . In the second setup, we do not change the error distributions but we violate Condition (C3) that requires two partial pure children per shared latent node. In this experiment, we do not make any sparsity assumptions on the mixing matrix G V, L .

Figure 8 shows the results of our experiments. As expected, we see in (a) and (b) that identifyability of the joint distribution fails if we do not require pairwise different error distributions. Recovering the joint distribution still works well in the second setup where we violate the partial pure children

(a)

(b)

(c)

3.5

0.08

Assump. satisfied

2.0

Assump. satisfied

3.0

scoreB

Sameerrors

Median scoreA

Sameerrors

2.5

0.06

No pure children

1.5

No pure children

2.0

1.5

Median

0.04

1.0

1.0

Assump. satisfied

0.02

0.5

0.5

Sameerrors

0.0

No pure children

0.00

0.0

10

3

10

4

10°

3

10

4

10

3

10

4

Samplesize

Sample size

Samplesize

conditions. However, identifying the shared latent graph is impossible in this setup, as we explained in Appendix C. This is supported by the experimental results displayed in Figure 8 (c), where we can see that recovery of the shared latent graph does not work when the partial pure children assumption is not satisfied.

G Future Work

We see many directions for future work, which include the following.

Our work and algorithms rely on linear ICA. It would be interesting to study a more direct approach to recover the joint distribution and the causal graph. This might potentially be done by testing certain constraints implied by the model similar as the developments in the LiNGAM literature; see e.g. Shimizu et al. (2011) and Wang and Drton (2020).

Our results require non-Gaussianity and that both the latent structural equation model and the mixing functions are linear. We consider the linear setup as a basis for any subsequent study of nonlinear cases. For example, recent advances in non-linear ICA allow identifiability of up to linear transformations, see e.g. Khemakhem et al. (2020), Buchholz et al. (2022) and Roeder et al. (2021). Identifiability of a causal representation might then be obtained from identifiability results for the linear model.

Our sufficient condition for identifiability of the shared latent graph requires two partial pure children per shared latent node. In this regard, it would also be interesting to study necessary conditions; c.f. our discussion in Appendix C.

This work focused purely on the observational case. However, considering interventional data can be expected to permit relaxing some conditions in both of the key steps, i.e., recovering the joint distribution and the shared latent graph. For example, recent work shows that interventional data allow for identification of the latent graph without sparsity constraints in a single-domain setup (Squires et al., 2023; Ahuja et al., 2023). Extending this to the multi-domain setup is an interesting problem for future work.

It would be interesting to study the statistical properties of our setup such as theoretical bounds on the accuracy of recovering the matrices B and A L , L as well as developing algorithms that meet these bounds. Currently, our adapted algorithms for finite samples determine the rank of a matrix by using a threshold for singular values. The algorithms depend on the choice of this parameter and it would be worth studying optimal choices. Moreover, one might consider different methods for determining the rank of a matrix.

There might be different matching strategies of the estimated error distributions in the finite sample setting. For example, instead of matching pairwise consistently across all domains as we propose in Appendix D, one might find an optimal matching by solving a linear program.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the setting of unsupervised learning where observations belong to several domains, and we only observe the marginal distribution of each domain. A set of latent variables generates the observations, where a subset of latents are shared across domains. The authors provide the first identification results in this setting, assuming that the latents follow a linear SCM, and the observations are an injective linear transformation of the latents. With this model, identifying the (unobserved) joint distribution of the observations equates to identifying the latents. The mapping between the exogenous noises and the observations, as well as the distributions over the exogenous noises, are identified up to signed block permutation. With additional conditions, the authors also identify the causal graph for the latents up to a signed permutation consistent with the topological ordering of the latents. The authors validate their claims with a synthetic data experiment.\n\nSTRENGTHS:\nThis is a strong paper that provides the first identifiability results on multiple-domain unsupervised learning where the joint distribution of the observed variables is unobserved. These results are impactful, since this problem setup is well-studied and has practical applications in single-cell biology. Existing approaches are significantly limited by their lack of identifiability, so this paper makes a valuable contribution.\n\nThe writing style is rigorous, and definitions, assumptions, and results are explained precisely.\n\nWEAKNESSES:\nThis paper could be improved with more context on how they are extending existing identification results to achieve theirs. Currently, the authors mention which existing results are being used, but do not provide an intuitive description on why they need to be extended, and how they do so.\n\nThe notation could be improved. Capital letters are used to denote matrices, probability measures, vector- and scalar-valued random variables, sets of nodes, and sets of edges. It would improve readability if you used font styles (e.g. lower-case bold for vectors) to differentiate them.\n\nQUESTIONS:\nThis paper makes two contributions. The first is to extend identifiable single-domain linear ICA to the multi-domain setting. The second is to extend single-domain graph identifiability to the multi-domain setting. In both cases, can you intuitively describe what about the multi-domain setting prevents the existing results from holding automatically?\n\nFor example, if there are no shared latents, the joint can be identified by trivially applying linear ICA separately to each domain. What is the difficulty imposed by a subset of latents being shared, and how is this circumvented?\n\nLIMITATIONS:\nThe authors made it explicit that this work is primarily about identifiability, and less about scalable algorithms and evaluation on realistic datasets.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address unpaired multi-domain causal representation. In detail, the authors learn the representations of the observed data from different domains that consist of causally. To achieve this, the authors consider the data generation process where the relationship between the latent variables is linear. Based on this generation process, they prove that the joint distribution of observed data and the shared causal structures of latent variables are identifiable.\n\nSTRENGTHS:\nThe authors investigate the causal discovery with latent variables from different domains.\n\nWEAKNESSES:\n1.\tThere are several works about causal discovery with latent variables under linear and multi-domain case like [1], which also considers the shared causal structure among latent variables and provide identification guarantees. It is suggested that the authors should discuss these works.  \n2.\tMoreover, the authors discuss several works about domain translation between unpaired data and claim that none of these works have rigorous identifiability guarantees. However, Multi-domain image generation, image translation, and domain adaptation belong to the proposed setting, and [2][3] have addressed the multi-domain causal representation learning problem recently. And the authors do not consider these works. It is noted that [2][3] considers the multi-domain causal representation learning with nonlinear transformation, which seems to be more general than the proposed setting.  \n3.\tAs for the identification of joint distributions, it is not clear why the identification of $l, B$, and $P$ can identify the joint distributions of observed data from multi-domains.   \n4.\tIn section 3, the authors assume that the distribution of errors is non-Gaussian for the identification of linear ICA by not allowing asymmetric distribution. But some distribution like the Laplace distribution is symmetric and they can also satisfy the identification of linear ICA.\n5.\tAccording to this paper, the authors consider the structure of latent variables to be linear but flexible. In the simulation experiment, the authors only consider three shared latent variables, it is suggested that the authors should consider more latent variables and different structures.  \n6.\tBesides, it is suggested that the authors should consider more compared methods and employ other metrics like recall, and precisions to evaluate the performance of causal discovery.  \n\n[1] Causal Discovery with Multi-Domain LiNGAM for Latent Factors  \n[2] Multi-domain image generation and translation with identifiability guarantees  \n[3] partial disentanglement for domain adaptation\n\nQUESTIONS:\nPlease refer weaknesses\n\nLIMITATIONS:\nPlease refer weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper considers causal representation learning from unpaired multi-domain data, with latent variables both shared and specific to domains.\n- Its key contribution is a new identifiability result for linear causal models with non-Gaussian noise, linear mixing function, and a number of other assumptions.\n- In addition, the authors develop a practical representation learning algorithm for this setting and demonstrate it on toy data.\n\nI've read the authors' rebuttal. They have addressed my concerns adequately.\n\nSTRENGTHS:\n- Causal representation learning is an interesting, relevant, and mostly unsolved problem.\n- The setting considered here (observational unpaired multi-domain data) is practical and well-motivated from single-cell biology applications.\n- The identifiability result is, to the best of my knowledge, novel, and substantially different from existing results.\n- As far as I can tell, it is also correct, though due to review overload I have not been able to check the proofs properly.\n- The paper is extraordinarily well-written. The authors manage to be precise, yet still provide intuitive explanations.\n\nWEAKNESSES:\n- The contribution made here has only one real weakness, and that is the host of strong assumptions underlying the identifiability result: 1D causal variables, linear causal model, no causal effects from shared to domain-specific latents, non-symmetric error distributions, different error distributions for each variable, linear mixing function, full-rank mixing function, observed variables include sufficient \"partially pure children\", and the list goes on. To put it bluntly, this list makes me wonder if this identifiability result present progress on the road to algorithms that work in practice on interesting real-world datasets.\n    - Of course, strong statements such as CRL identifiability require strong inputs, but these need not be in the form of model assumptions, they could also come from the data side. Perhaps it is a bit out of scope for this paper, but I would be curious if the availability of *interventional* data or some other form of auxiliary data would allow the relaxation of some of these model assumptions.\n    - While the authors do a good job in providing an intuition for why these assumptions are needed, I would like to know if there are any real-world problems that satisfy them all. This is partially discussed for single-cell data and a few of these assumptions, but could the authors provide a more complete example that ideally satisfies all assumptions?\n- There are no experiments to speak of, though I also don't think that all papers need experiments.\n\nQUESTIONS:\n- See above.\n- Very minor suggestion: there are a few instances of `\\citet{}` that should be a `\\citep{}`. It speaks for the quality of the writing that I can't think of any other comments here.\n\nLIMITATIONS:\n- The paper is very clear about the (many) assumptions in the theory. It also openly acknowledges the limitations of the experiments.\n- I do not see any particular need for an extensive discussion of societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This work tackles the problem of learning the latent causal structure from multiple unpaired domains. Under a linear non-Gaussian condition, this work presents the identifiability guarantees for the joint distribution over the domains and the causal structure within the shared latent partition. Synthetic data experiments are presented to validate the theory.

STRENGTHS:
1. The problem is well-motivated and timely. Unpaired data are prevalent in the wild, and this work provides a rigorous treatment as the initial step to leverage such data in a principled manner.
2. The paper is nicely written, and the theoretical analysis is clearly articulated with sufficient explanations.
3. The theoretical techniques are clearly explained. Connections and attributions to prior work are appropriately introduced, which aids the assessment of this paper’s technical contribution.

WEAKNESSES:
1. The linear assumptions: practical multi-domain (modal) data-generating processes are often highly nonlinear, e.g., images and text. The applicability of the linear assumption may not be as appealing.
2. The heterogeneous noise distributions: pairwise distinct exogenous distributions appear a strong assumption to me and can potentially oversimplify the technical challenge. I would be interested in learning about the necessity of such an assumption.

QUESTIONS:
I would like to learn about the authors' response to the weaknesses listed above, which may give me a clearer perspective on the paper's contribution.

LIMITATIONS:
Please see the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07