Title: Learning Linear Causal Representations from Interventions under General Nonlinear Mixing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal Representation Learning

Interventional data

Gaussian Structural Causal models
Abstract: We study the problem of learning causal representations from unknown, latent interventions in a general setting, where the latent distribution is Gaussian but the mixing function is completely general. We prove strong identifiability results given unknown single-node interventions, i.e., without having access to the intervention targets. This generalizes prior works which have focused on weaker classes, such as linear maps or paired counterfactual data. This is also the first instance of identifiability from non-paired interventions for deep neural network embeddings and general causal structures. Our proof relies on carefully uncovering the high-dimensional geometric structure present in the data distribution after a non-linear density transformation, which we capture by analyzing quadratic forms of precision matrices of the latent distributions. Finally, we propose a contrastive algorithm to identify the latent variables in practice and evaluate its performance on various tasks.

Learning Linear Causal Representations from Interventions under General Nonlinear Mixing

Abstract

We study the problem of learning causal representations from unknown, latent interventions in a general setting, where the latent distribution is Gaussian but the mixing function is completely general. We prove strong identifiability results given unknown single-node interventions, i.e., without having access to the intervention targets. This generalizes prior works which have focused on weaker classes, such as linear maps or paired counterfactual data. This is also the first instance of identifiability from non-paired interventions for deep neural network embeddings and general causal structures. Our proof relies on carefully uncovering the highdimensional geometric structure present in the data distribution after a non-linear density transformation, which we capture by analyzing quadratic forms of precision matrices of the latent distributions. Finally, we propose a contrastive algorithm to identify the latent variables in practice and evaluate its performance on various tasks.

1 Introduction

Modern generative models such as GPT-4 [59] or DDPMs [26] achieve tremendous performance for a wide variety of tasks [7]. They do this by effectively learning high-level representations which map to raw data through complicated non-linear maps , such as transformers [89] or diffusion processes [79]. However, we are unable to reason about the specific representations they learn, in particular they are not necessarily related to the true underlying data generating process. Besides their susceptibility to bias [20], they often fail to generalize to out-of-distribution settings [5], rendering them problematic in safety-critical domains. In order to gain a deeper understanding of what representations deep generative models learn, one line of work has pursued the goal of causal representation learning (CRL) [75]. CRL aims to learn high-level representations of data while simultaneously recovering the rich causal structure embedded in them, allowing us to reason about them and use them for higher-level cognitive tasks such as systematic generalization and planning. This has been used effectively in many application domains including genomics [52] and robotics [43, 94].

A crucial primitive in representation learning is the fundamental notion of identifiability [37, 70], i.e., the question whether a unique (up to tolerable ambiguities) model can explain the observed data. It is well known that because of non-identifiability, CRL is impossible in general settings in the absence of inductive biases or additional information [29, 50]. In this work, we consider additional information

in the form of interventional data [75]. It is common to have access to such data in many application domains such as genomics and robotics stated above (e.g. [17, 58, 57]). Moreover, there is a pressing need in such safety-critical domains to build reliable and trustworthy systems, making identifiable CRL particularly important. Therefore, it's important to study whether we can and also how to perform identifiable CRL from raw observational and interventional data. Here, identifiability opens the possibility to provably recover the true representations with formal guarantees. Meaningfully learning such representations with causal structure enables better interpretability, allows us to reason about fairness, and helps with performing high-level tasks such as planning and reasoning.

In this work, we study precisely this problem of causal representation learning in the presence of interventions. While prior work has studied simpler settings of linear or polynomial mixing [84, 88, 2, 71, 11], we allow for general non-linear mixing, which means our identifiability results apply to complex real-world systems and datasets which are used in practice. With our results, we make progress on fundamental questions on interventional learning raised by [75].

Concretely, we consider a general model with latent variables Z and observed data X generated as X = f ( Z ) where f : R d → R d ′ is an arbitrary non-linear mixing function. We assume Z satisfies a Gaussian structural equation model (SEM) which is unknown and unobserved. A Gaussian prior is commonly used in practice (which implies a linear SEM over Z ) and further, having a simple model for Z allows us to learn useful representations, generate data efficiently, and explore the latent causal relationships, while the non-linearity of f ensures universal approximability of the model. We additionally assume access to interventional data X ( i ) = f ( Z ( i ) ) for i ∈ I where Z ( i ) is the latent under an intervention on a single node Z t i . Notably, we allow for various kinds of interventions, we don't require knowledge of the targets t i , and we don't require paired data, (i.e., we don't need counterfactual samples from the joint distribution ( X,X ( i ) ) , but only their marginals). Having targets or counterfactual data is unrealistic in many practical settings but many prior identifiability results require them, so eliminating this dependence is a crucial step towards CRL in the real world.

Contributions Our main contribution is a general solution to the identifiability problem that captures practical settings with non-linear mixing functions (e.g. deep neural networks) and unknown interventions (since the targets are latent). We study both perfect and imperfect interventions and additionally allow for shift interventions, where perfect interventions remove the dependence of the target variable from its parents, while imperfect interventions (also known as soft interventions) modify the dependencies. Below, we summarize our main contributions:

We show identifiability of the full latent variable model, given non-paired interventional data with unknown intervention targets. In particular, we learn the mixing, the targets and the latent causal structure.

Compared to prior works that have focused on linear/polynomial f , we allow non-linear f which encompasses representations learned by e.g. deep neural networks and captures complex real-world datasets. Moreover, we study both imperfect (also called soft) and perfect interventions, and always allow shift interventions.

We construct illustrative counterexamples to probe tightness of our assumptions which suggest directions for future work.

We propose a novel algorithm based on contrastive learning to learn causal representations from interventional data, and we run experiments to validate our theory. Our experiments suggest that a contrastive approach, which so far has been unexplored in interventional CRL, is a promising technique going forward.

2 Related work

Causal representation learning [75, 74] has seen much recent progress and applications since it generalizes and connects the fields of Independent Component Analysis [13, 28, 30], causal inference [81, 61, 63, 64, 82] and latent variable modeling [40, 3, 78, 96, 32]. Fundamental to this approach is the notion of identifiability [36, 15, 93]. Due to non-identifiability in general settings without inductive biases [29, 50], prior works have approached this problem from various angles - using additional auxiliary labels [36, 27, 6, 76]; by imposing sparsity [75, 42, 54, 103]; or by restricting the function classes [41, 9, 104, 22]. See also the survey [32]. Moreover, a long line of works have proposed practical methods for CRL (which includes causal disentanglement as a special case),

(a) No interventions

(b) An imperfect intervention

(c) A perfect intervention

Z1

Z3

i

(i)

4

2

4

Nonlinear

Nonlinear

Nonlinear

mixing f

mixing f

mixing f

X1

X2

X3

X4

X5

(i)

i

1

[19, 95, 16, 97, 44, 14] to name a few. It's worth noting that most of these approaches are essentially variants of the Variational Autoencoder framework [39, 68].

Of particular relevance to this work is the setting when interventional data is available. We first remark that the much simpler case of fully observed variables (i.e., no latent variables), has been studied in e.g. [23, 62, 83, 34, 18] (see also the survey [82]). In this work, we consider the more difficult setting of structure learning over latent variables, which have been explored in [46, 42, 6, 104, 1, 84, 88, 2, 71, 11, 72]. Among these, [46] assumes that the intervention targets are known, [42] specifically consider instance-level pre- and post- interventions for time-series data and [6, 104, 1] assumes access to paired counterfactual data. In contrast, we assume unknown targets and work in general settings with non-paired interventional data, which is important in various real-world applications [85, 99]. The work [48] require several graphical restrictions on the causal graph and also require 2 d interventions, while we make no graphical restrictions and only require d interventions (which we also show cannot be improved under our assumptions). [71, 11, 84, 88] consider linear mixing functions f whereas we study general non-linear f . Finally, [2] consider polynomial mixing in the more restricted class of deterministic do-interventions. Several concurrent works study related settings [100, 35, 65, 45, 92]. For a more detailed comparison to the related works, we refer to Appendix C.

Our proposed algorithm is based on contrastive learning. Contrastive learning has been used in other contexts in this domain [31, 27, 55, 104, 91] (either in the setting of time-series data or paired counterfactual data), however the application to non-paired interventional settings is new to the best of our knowledge.

Notation In this work, we will almost always work with vectors and matrices in d dimensions where d is the latent dimension, and we will disambiguate when necessary. For a vector v , we denote by v i its i -th entry. Let Id denote the d × d identity matrix with columns as the standard basis vectors e 1 , . . . , e d . We denote by N ( µ, Σ) the multivariate normal distribution with mean µ and covariance Σ . For a set C , let U ( C ) denote the uniform distribution on C . For two random variables X,X ′ , we write X D = X ′ if their distributions are the same. We denote the set { 1 , . . . , d } by [ d ] . For permutation matrices we use the convention that ( P ω ) ij = 1 j = ω ( i ) for ω ∈ S d . We use standard directed graph terminology, e.g. edges, parents. When we use the term 'non-linear mixing' in this work, we also include linear mixing as a special case.

3 Setting: Interventional Causal Representation Learning

We now formally introduce our main settings of interest and the required assumptions. We assume that there is a latent variable distribution Z on R d and an observational distribution X on R d ′ given by X = f ( Z ) where d ≤ d ′ and f is a non-linear mixing. This encapsulates the most general definition of a latent variable model. As the terminology suggests, we observe X in real-life, e.g. images of cats and Z encodes high-level latent information we wish to learn and model, e.g. Z could indicate orientation and size.

Assumption 1 (Nonlinear f ) . The non-linear mixing f is injective, differentiable and embeds into R d ′ .

Such an assumption is standard in the literature. Injectivity is needed in order to identify Z from X because otherwise we may have learning ambiguity if multiple Z s map to the same X . Differentiability is needed to transfer densities. Note that Assumption 1 allows for a large class of complicated nonlinearities and we discuss in Appendix E why proving results for such a large class of mixing functions is important.

̸

Next, we assume that the latent variables Z encode causal structure which can be expressed through a Structural Causal Model (SCM) on a Directed Acyclic Graph (DAG). We focus on linear SCMs with an underlying causal graph G on vertex set [ d ] , encoded by a matrix A through its non-zero entries, i.e., there is an edge i → j in G iff A ji = 0 .

Assumption 2 (Linear Gaussian SCM on Latent Variables) . The latent variables Z follow a linear SCM with Gaussian noise, so

where D is a diagonal matrix with positive entries, A encodes a DAG G and ϵ ∼ N (0 , Id) .

For an illustration of the model, see Fig. 1a. It is convenient to encode the coefficients of linear SCMs through the matrix B = D -1 / 2 (Id -A ) . Then Z = B -1 ϵ and both D and A can be recovered from B , indeed the diagonal entries of B agree with the entries of D -1 / 2 . Note that we can and will always assume that the entries of D are positive since ϵ D = -ϵ .

Assuming that the latent variables follow a linear Gaussian SCM is certainly restrictive but nevertheless a reasonable approximation of the underlying data generating process in many settings. The Gaussian prior assumption is also standard in latent variable modeling and enables efficient inference for downstream tasks, among other advantages. Importantly, under the above assumptions, our model has infinite modeling capacity [53, 86, 33], so they're able to model complex datasets such as images and there's no loss in representational power.

The goal of representation learning is to learn the non-linear mixing f and the high-level latent distribution Z from observational data X . In causal representation learning, we wish to go a step further and model Z as well by learning the parameters B which then lets us easily recover A,D and the causal graph G . For general non-linear mixing, this model is not identifiable and hence we cannot hope to recover Z , but in this work we use additional interventional data, similar to the setups in recent works [2, 84, 88].

Assumption 3 (Single node interventions) . We consider interventional distributions Z ( i ) for i ∈ I which are single node interventions of the latent distribution Z , i.e., for each intervention i there is a target node t i and we change the assigning equation of Z ( i ) t i to

̸

while leaving all other equations unchanged. We assume that A ( i ) t i ,k = 0 only if k → t i in G , i.e., no parents are added and η ( i ) denotes a shift of the mean.

For an illustration, see Fig. 1b. An intervention on node t i has no effect on the non-descendants of t i , but will have a downstream effect on the descendants of t i . In particular, A is modified so that the weight of the edge k → t i could potentially be changed if it exists already but no new incoming edges to t i may be added. Also, the noise variable ϵ t i is also allowed to be modified via a scale intervention as well as a shift intervention. Note that [84, 2] do not allow shift interventions, whereas we do.

Again, this can be written concisely as Z ( i ) = ( B ( i ) ) -1 ( ϵ + η ( i ) e t i ) where B ( i ) = ( D ( i ) ) -1 / 2 (Id -A ( i ) ) . Let r ( i ) denote the row t i of B ( i ) , then we can write B ( i ) = B -e t i ( B ⊤ e t i -r ( i ) ) ⊤ . We

D

assume that interventions are non-trivial, i.e., Z ( i ) = Z . In our model, we observe interventional distributions X ( i ) = f ( Z ( i ) ) for various interventions i ∈ I .

̸

̸

̸

Definition 1 (Intervention Types) . We call an intervention perfect (or stochastic hard) if A ( i ) t i · = 0 , i.e., we remove all connections to former parents and potentially change variance and mean of the noise variable. Note that for nodes without parents, either D ( i ) t i t i = D t i t i or η ( i ) = 0 so that the intervention is non-trivial. We call an intervention a pure noise intervention if A ( i ) t i · = A t i · and A t i · = 0 (i.e., node t i has at least one parent). In other words, a pure noise intervention targets a

̸

node with parents by changing only the noise distribution through a change of variance (encoded in D ) or a mean shift (encoded in η ).

We call a single-node intervention imperfect if it is not perfect (some prior works call it a soft intervention). Note that perfect interventions are never of pure noise type because they necessarily change the relation to the parents. For an illustration of perfect and imperfect interventions, see Fig. 1c, Fig. 1b respectively. Finally, we require an exhaustive set of interventions.

Assumption 4 ((Coverage of Interventions)) . All nodes are intervened upon by at least one intervention, i.e., { t i : i ∈ I } = [ d ]

This assumption was also made in the prior works [84, 2, 88]. When the interventions don't cover all the nodes, we have non-identifiability as described in Section 4 and Appendix D. We also extensively discuss that none of our other assumptions can simply be dropped in these sections.

Remark 1. · Our theory also readily extends to noisy observations, i.e., X = f ( Z ) + ν where ν is independent noise. In this case, we first denoise via a standard deconvolution argument [36, 42] and then apply our theory.

Similar to the results in [84] we can assume completely unknown intervention targets, i.e., there might be multiple interventions targeting the same node, and we do not need to know the partition. We only require coverage of all nodes. In contrast to their work we assume that we know which dataset corresponds to the observational distribution, but we expect that this restriction can be removed.

To simplify the notation, it is convenient to use B (0) = B , η (0) = 0 for the observational distribution. We also define ¯ I = I ∪ { 0 } . Then, all information about the latent variable distributions Z ( i ) and observed distributions X ( i ) are contained in (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) .

4 Main Results

We can now state our main results for the setting introduced above.

Theorem 1. Suppose we are given distributions X ( i ) generated using a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) such that Assumptions 1-4 hold and such that all interventions i are perfect. Then the model is identifiable up to permutation and scaling, i.e., for any model (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ ¯ I , ˜ f ) that generates the same data, the latent dimension d agrees and there is a permutation ω ∈ S d (and associated permutation matrix P ω ) and an invertible pointwise scaling matrix Λ ∈ Diag( d ) such that

This in particular implies that

and we can identify the causal graph G up to permutation of the labels.

This result says that for the interventional model as described in the previous section, we can identify the non-linear map f , the intervention targets t i , the parameter matrices B,D,A up to permutations P ω and diagonal scaling Λ . Moreover, we can recover the shifts η ( i ) exactly and also the underlying causal graph G up to permutations.

Remark 2. Identifiability and recovery up to permutation and scaling is the best possible for our setting. This is because the latent variables Z are not actually observed, which means we cannot (and in fact don't need to) resolve permutation and scaling ambiguity without further information about Z . See [84, Proposition 1] for the short proof.

When we drop the assumption that the interventions are perfect, we can still obtain a weaker identifiability result. Define ≺ G to be the minimal partial order on [ d ] such that i ≺ G j if ( i, j ) is an edge in G , i.e., i ≺ G j if and only if i is an ancestor of j in G . Note that any topological ordering of G is compatible with the partial order ≺ G . Then, our next result shows that under imperfect interventions, we can still recover the partial order ≺ G .

Theorem 2. Suppose we are given the distributions X ( i ) generated using a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) with causal graph G such that the Assumptions 1-4 hold and none of the interventions is a pure noise intervention. Then for any other model (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ ¯ I , ˜ f ) with causal graph ˜ G generating the same observations the latent dimension d agrees and there is a permutation ω ∈ S d such that ˜ t i = ω ( t i ) and i ≺ ˜ G j iff ω ( i ) ≺ G ω ( j ) , i.e., ≺ G can be identified up to a permutation of the labels.

Remark 3. We emphasize that neither the full graph nor the coefficients B ( i ) or the latent variables Z i are identifiable in this setting, even for linear mixing functions.

Theorem 1 and Theorem 2 generalize the main results of [84] which assume linear f while we allow for non-linear f . The key new ingredient of our work is the following theorem, which shows identifiability of f up to linear maps.

Theorem 3. Assume that X ( i ) is generated according to a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) such that the Assumptions 1-4 hold. Then we have identifiability up to linear transformations, i.e., if (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) , ˜ f ) generates the same observed distributions ˜ f ( ˜ Z ( i ) ) D = X ( i ) , then their latent dimensions d agree and there is an invertible linear map T such that

The proof of this theorem is deferred to Appendix A. In Appendix B we then review the results of [84] and show how their results can be extended to obtain our main results, Theorem 1 and Theorem 2. Now we provide some intuition for the proofs of the main theorems.

Proof intuition The recent work [84] studies the special case when f is linear, and the proof is linear algebraic. In particular, they consider row spans of the precision matrices of X ( i ) , project them to certain linear subspaces and use those subspaces to construct a generalized RQ decomposition of the pseudoinverse of the linear mixing matrix f . However, once we are in the setting of non-linear f , such an approach is not feasible because we can no longer reason about row spans of the precision matrices of X , which have been transformed non-linearly thereby losing all linear algebraic structure. Instead, we take a statistical approach and look at the log densities of the X ( i ) . By choosing a Gaussian prior, the log-odds ln p ( i ) X ( x ) -ln p (0) X ( x ) of X ( i ) with respect to X (0) can be written as a quadratic form of difference of precision matrices, evaluated at non-linear functions of x . For simplicity of this exposition, ignore terms arising from shift interventions and determinants of covariance matrices. Then, the log-odds looks like θ ( x ) = f -1 ( x ) ⊤ (Θ ( i ) -Θ (0) ) f -1 ( x ) , where Θ ( i ) is the precision matrix of Z ( i ) .

At this stage, we again shift our viewpoint to geometric and observe that Θ ( i ) -Θ (0) has a certain structure. In particular, for single-node interventions, it has rank at most 2 and for source node targets, it has rank 1 . This implies that the level set manifolds of the quadratic forms θ ( x ) also have a certain geometric structure in them, i.e., the DAG leaves a geometric signature on the data likelihood. We exploit this carefully and proceed by induction on the topological ordering until we end up showing that f can be identified up to a linear transformation, which is our main Theorem 3. Here, the presence of shift interventions adds additional complexities, and we have to generalize all of our intermediate lemmas to handle these. Once we identify f up to a linear transformation, we can apply the results of [84] to conclude Theorems 1 and 2.

On the optimality and limitations of the assumptions We make a few brief remarks on our assumptions, deferring to Appendix D a full discussion and the technical construction of several illustrative counterexamples.

Number of interventions: For our main results, Theorems 1 and 2, we assume that there are at least d interventions (Assumption 4). This cannot be weakened even for linear mixing functions [84]. In addition, we also show in Fact 1 that for the linear identifiability proved in Theorem 3, d -2 interventions are not sufficient. Thus, the required number of interventions is tight in Theorems 1, 2 and is tight up to at most one intervention in Theorem 3.

Intervention type: In the setting of imperfect interventions, the weaker identifiability guarantees in Theorem 2 cannot be improved even when the mixing is linear [84, Appendix C]. We also show in Fact 2 that if we drop the condition that interventions are not pure noise interventions, we have non-identifiability. Concretely, we show that when all interventions

are of pure noise type, any causal graph is compatible with the observations. Finally, in contrast to the special case of linear mixing, we show in Lemma 8 that we need to exclude non-stochastic hard interventions (i.e., do( Z i = z i ) ) for identifiability up to linear maps.

Distributional assumptions: We assume Gaussian latent variables, while allowing for very flexible mixing f . This model has universal approximation guarantees and is moreover used ubiquitously in practice. While the result can potentially be extended to more general latent distributions (e.g., exponential families), we additionally show in Lemma 9 that the result is not true when making no assumption on the distribution of ϵ .

5 Experimental methodology

In this section, we explain our experimental methodology and the theoretical underpinning of our approach. Our main experiments for interventional causal representation learning focus on a method based on contrastive learning. We train a deep neural network to learn to distinguish observational samples x ∼ X (0) from interventional samples x ∼ X ( i ) . Additionally, we design the last layer of the model to model the log-likelihood of a linear Gaussian SCM. Due to representational flexibility of deep neural networks, we will in principle learn the Bayes optimal classifier after optimal training, which we show is related to the underlying causal model parameters. Accordingly, with careful design of the last layer parametric form, we indirectly learn the parameters of the underlying causal model. Similar methods have been used for time-series data or multimodal data [27, 32] but to the best of our knowledge, the contrastive learning approach to interventional learning is novel.

Denote the probability density of x ∼ X ( i ) (resp. z ∼ Z ( i ) ) by p ( i ) X ( x ) (resp. p ( i ) Z ( z ) ). The next lemma describes the log-odds of a sample x coming from the interventional distribution X ( i ) as opposed to the observational distribution X (0) . We focus on the identifiable case (see Theorem 1) and therefore only consider perfect interventions. As per the notation in Section 3 and Appendix A.1, let s ( i ) denote the row t i of B (0) and let η ( i ) , λ i denote the magnitude of the shift and scale intervention respectively.

Lemma 1. When we have perfect interventions, the log-odds of a sample x coming from X ( i ) over coming from X (0) is given by

for a constant c i independent of x .

The proof is deferred to Appendix F. The form of the log-odds suggests considering the following functions

where h ( · , θ ) denotes a neural net parametrized by θ , parameters w ( i ) are the rows of a matrix W and α i , β i , γ i are learnable parameters. Note that the ground truth parameters minimize the following cross entropy loss

Note that (compare (6) and (7)) W should learn B = D -1 / 2 (Id -A ) thus its off-diagonal entries should form a DAG. To enforce this we add the NOTEARS regularizer [102] given by R NOTEARS ( W ) = tr exp( W 0 ◦ W 0 ) -d (see Appendix F.3) where W 0 equals W with the main diagonal zeroed out. We also promote sparsity by adding the l 1 regularization term R REG ( W ) = ∥ W 0 ∥ 1 . Thus, the total loss is given by

for hyperparameters τ 1 and τ 2 . Our identifiability result Theorem 1 implies that when we assume that the neural network has infinite capacity, the loss in (9) is minimized, τ 1 is large and τ 2 small, and we learn Gaussian latent variables h ( X ( i ) , θ ) , then we recover the ground truth latent variables up to the

10

20

30

d

5

10

SHD

10

20

30

d

0.995

1.000

AUROC

10

20

30

d

0.925

0.950

0.975

MCC

10

20

30

d

0.90

0.95

R

2

tolerable ambiguities of labeling and scale, i.e., h recovers f -1 and W recovers B up to permutation and scale. Thus, we estimate the latent variables using ˆ Z = h ( X,θ ) and estimate the DAG using W 0 . Full details of the practical implementation of our approach are given in Appendix H.

Our experimental setup is similar to [84, 2], we consider d interventions with different targets (our theory holds in full generality) and therefore, we can arbitrarily assign t i = i based on the intervention index i which removes the permutation ambiguity. We focus on non-zero shifts because the cross entropy loss together with the quadratic expression for the log-odds results in a non-convex output layer which makes it hard to find the global loss minimizer, as we will describe in more detail in Appendix G.1. We emphasize that this is no contradiction to the theoretical results stated above. Even when there is no shift intervention the latent variables are identifiable, but our algorithm often fails to find the global minimizer of the loss (9) due to the non-convex loss landscape. For the sake of exposition and to set the stage for future work, we also briefly describe an approach via Variational Autoencoders (VAE). VAEs have been widely used in causal representation learning and while feasible in interventional settings, they are accompanied by certain difficulties, which we detail and suggest how to overcome in Appendix F.4.

6 Experiments

In this section, we implement our approach on synthetic data and image data. Complete details (architectures, hyperparameters) are deferred to Appendix H. Additional experiments investigating the effect of varsortability [66] and the noise distribution can be found in Appendix G.

Data generation For all our experiments we use Erdös-Rényi graphs, i.e., we add each undirected edge with equal probability p to the graph and then finally orient them according to some random order of the nodes. We write ER( d, k ) for the Erdös-Rényi graph distribution on d nodes with kd expected edges. For a given graph G we then sample edge weights from U ( ± [0 . 25 , 1 . 0]) and a scale matrix D . For simplicity we assume that we have n samples from each environment i ∈ ¯ I . We only consider the setting where each node is intervened upon once and thus the latent dimension is also known. We consider three types of mixing functions. First, we consider linear mixing functions where we sample all matrix entries i.i.d. from a Gaussian distribution. Then, we consider non-linear mixing functions that are parametrized by MLPs with three hidden layers which are randomly initialized, and have Leaky ReLU activations. Finally, we consider image data as described in [2]. Pairs of latent variables ( z 2 i +1 , z 2 i +2 ) describe the coordinates of balls in an image and the non-linearity f is the rendering of the image. The image generation is based on pygame [77]. A sample image can be found in Figure 3.

Evaluation Metrics We evaluate how well we can recover the ground truth latent variables and the underlying DAG. For the recovery of the ground truth latents we use the Mean Correlation Coefficient (MCC) [37, Appendix A.2]. For the evaluation of the learned graph structure we use the Structural Hamming Distance (SHD) where we follow the convention that we count the number of edge differences of the directed graphs (i.e., an edge with wrong orientation counts as two errors). Since the scale of the variables is not fixed the selection of the edge selection threshold is slightly delicate. Thus, we use the heuristic where we match the number of selected edges to the expected number

of edges. As a metric that is independent of this thresholding procedure, we also report the Area Under the Receiver Operating Curve (AUROC) for the edge selection.

Methods We implement our contrastive algorithm as explained in Section F where we use an MLP (with Leaky ReLU nonlinearities) for the function h for the linear and nonlinear mixing functions and a very small convolutional network for the image dataset, which are termed 'Contrastive'. For linear mixing function we also consider a version of the contrastive algorithm where h is a linear function, termed 'Contrastive Linear'. As baselines, we consider a variational autoencoder with latent dimension d and also the algorithm for linear disentanglement introduced in [84]. Since the variational autoencoder does not output a causal graph structure we report the result for the empty graph here which serves as a baseline.

Results for linear f First, for the sake of comparison, we replicate exactly the setting from [84], i.e., we consider initial noise variances sampled uniformly from [2 , 4] and we consider perfect interventions where the new variance is sampled uniformly from [6 , 8] . We set d = 5 , d ′ = 10 , k = 3 / 2 , and n = 50000 . Results can be found in Table 1. The linear contrastive method identifies the ground truth latent variables up to scaling. The failure of the nonlinear method to recover the ground truth latents will be explained and further analyzed in Appendix G.1. Note that the nonlinear contrastive and the linear contrastive method recover the underlying graph better than the baseline.

Results for nonlinear f We sample all variances from U ([1 , 2]) (initial variance and resampling for the perfect interventions) and the shift parameters η ( i ) of the interventions from U ([1 , 2]) . Results can be found in Table 2. We find that our contrastive method can recover the ground truth latents and the causal structure almost perfectly, while the baseline for linear disentanglement cannot recover the graph or the latent variables (which is not surprising as the mixing is highly nonlinear). Also, training a vanilla V AE does not recover the latent variables up to a linear map as indicated by the R 2 scores. In Figure 2 we illustrate the dependence on the dimension d of our algorithm in this setting.

Results for image data Finally, we report the results for image data. Here, we generate the graph as before and consider variances sampled from σ 2 ∼ U ([0 . 01 , 0 . 02]) and shifts η ( i ) from U ([0 . 1 , 0 . 2]) (i.e., the shifts are still of order σ ). We exclude samples where one of the balls is not contained in the image which generates a slight model misspecification compared to our theory. Again we find that we recover the latent graph and the latent variables as detailed in Table 3.

7 Conclusion

In this work, we extend several prior works and show identifiability for a widely used class of linear latent variable models with non-linear mixing, from interventional data with unknown targets. Counterexamples show that our assumptions are tight in this setting and could only potentially be relaxed under other additional assumptions. We leave to future work to extend our results for other classes of priors, such as non-parametric distribution families, and also to study sample complexity and robustness of our results. We also proposed a contrastive approach to learn such models in practice and showed that it can recover the latent structure in various settings. Finally, we highlight that the results of our experiments are very promising and it would be interesting to scale up our algorithms to large-scale datasets such as [49].

Acknowledgments

We thank anonymous reviewers for useful comments and suggestions. We acknowledge the support of AFRL and DARPA via FA8750-23-2-1015, ONR via N00014-23-1-2368, and NSF via IIS-1909816, IIS-1955532. We also acknowledge the support of JPMorgan Chase & Co. AI Research. This work was also supported by the Tübingen AI Center.

References

K. Ahuja, J. S. Hartford, and Y . Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems , 35:15516-15528, 2022.

K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

S. Arora, R. Ge, Y. Halpern, D. M. Mimno, A. Moitra, D. A. Sontag, Y. Wu, and M. Zhu. A practical algorithm for topic modeling with provable guarantees. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings , pages 280-288. JMLR.org, 2013. URL http://proceedings.mlr.press/v28/arora13.html .

K. Bello, B. Aragam, and P. K. Ravikumar. DAGMA: Learning DAGs via m-matrices and a log-determinant acyclicity characterization. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https: //openreview.net/forum?id=8rZYMpFUgK .

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

J. Brehmer, P. de Haan, P. Lippe, and T. S. Cohen. Weakly supervised causal representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,

Advances in Neural Information Processing Systems , volume 35, pages 38319-38331. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/fa567e2b2c870f8f09a87b6e73370869-Paper-Conference.pdf .

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 , 2023.

S. Buchholz. Some remarks on identifiability of independent component analysis in restricted function classes. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=REtKapdkyI .

S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=DpKaP-PY8bK .

F. Castelletti and S. Peluso. Network structure learning under uncertain interventions. Journal of the American Statistical Association , pages 1-12, 2022.

Y. Chen, E. Rosenfeld, M. Sellke, T. Ma, and A. Risteski. Iterative feature matching: Toward provable domain generalization with logarithmic environments. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 1725-1736. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 0b5eb45a22ff33956c043dd271f244ea-Paper-Conference.pdf .

D. M. Chickering. Optimal structure identification with greedy search. J. Mach. Learn. Res. , 3:507-554, 2002. URL http://jmlr.org/papers/v3/chickering02b.html .

P. Comon. Independent component analysis, a new concept? Signal processing , 36(3): 287-314, 1994.

J. Cui, W. Huang, Y. Wang, and Y. Wang. Aggnce: Asymptotically identifiable contrastive learning.

A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research , 23(1):10237-10297, 2022.

N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648 , 2016.

A. Dixit, O. Parnas, B. Li, J. Chen, C. P. Fulco, L. Jerby-Arnon, N. D. Marjanovic, D. Dionne, T. Burks, R. Raychowdhury, B. Adamson, T. M. Norman, E. S. Lander, J. S. Weissman, N. Friedman, and A. Regev. Perturb-seq: Dissecting molecular circuits with scalable single-cell rna profiling of pooled genetic screens. Cell , 167(7):1853-1866, Dec 2016. ISSN 1097-4172 (Electronic); 0092-8674 (Print); 0092-8674 (Linking). doi: 10.1016/j.cell.2016.11.038.

F. Eberhardt, C. Glymour, and R. Scheines. On the number of experiments sufficient and in the worst case necessary to identify all causal relations among N variables. In UAI '05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, Edinburgh, Scotland, July 26-29, 2005 , pages 178-184. AUAI Press, 2005. URL https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu= 1&smnu=2&article_id=1235&proceeding_id=21 .

F. Falck, H. Zhang, M. Willetts, G. Nicholson, C. Yau, and C. C. Holmes. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems , 34, 2021.

E. Ferrara. Should ChatGPT be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738 , 2023.

J. L. Gamella, A. Taeb, C. Heinze-Deml, and P. Bühlmann. Characterization and greedy learning of gaussian structural causal models under unknown interventions, 2022.

L. Gresele, J. Von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, a new concept? Advances in Neural Information Processing Systems , 34, 2021.

A. Hauser and P. Bühlmann. Characterization and greedy learning of interventional markov equivalence classes of directed acyclic graphs. The Journal of Machine Learning Research , 13 (1):2409-2464, 2012.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

C. Heinze-Deml, J. Peters, and N. Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference , 6(2), 2018.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 3765-3773, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ d305281faf947ca7acade9ad5c8c818c-Abstract.html .

A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks , 12(3):429-439, 1999.

A. Hyvarinen, J. Karhunen, and E. Oja. Independent component analysis. Studies in informatics and control , 11(2):205-207, 2002.

A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

A. Hyvärinen, I. Khemakhem, and R. Monti. Identifiability of latent-variable and structuralequation models: from linear to nonlinear. arXiv preprint arXiv:2302.02672 , 2023.

I. Ishikawa, T. Teshima, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204.07415 , 2022.

A. Jaber, M. Kocaoglu, K. Shanmugam, and E. Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. Advances in neural information processing systems , 33:9551-9561, 2020.

Y. Jiang and B. Aragam. Learning latent causal graphs with unknown interventions. In Advances in Neural Information Processing Systems , 2023.

I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

I. Khemakhem, R. P. Monti, D. P. Kingma, and A. Hyvärinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ICA. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/ 2020/hash/962e56a8a0b0420d87272a682bfd1e53-Abstract.html .

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http: //arxiv.org/abs/1412.6980 .

D. P. Kingma and M. Welling. Auto-encoding variational bayes. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/ abs/1312.6114 .

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Learning latent causal graphs via mixture oracles. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 18087-18101, 2021. URL https://proceedings.neurips.cc/paper/2021/ hash/966aad8981dcc75b5b8ab04427a833b2-Abstract.html .

B. Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 649f080d8891ab4d4b262cb9cd52e69a-Abstract-Conference.html .

S. Lachapelle, P. Rodríguez, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. LacosteJulien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In B. Schölkopf, C. Uhler, and K. Zhang, editors, 1st Conference on Causal Learning and Reasoning, CLeaR 2022, Sequoia Conference Center, Eureka, CA, USA, 11-13 April, 2022 , volume 177 of Proceedings of Machine Learning Research , pages 428-484. PMLR, 2022. URL https://proceedings.mlr.press/v177/lachapelle22a.html .

T. E. Lee, J. A. Zhao, A. S. Sawhney, S. Girdhar, and O. Kroemer. Causal reasoning in simulation for structure and transfer learning of robot manipulation policies. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 4776-4782. IEEE, 2021.

S. Li, B. Hooi, and G. H. Lee. Identifying through flows for recovering latent representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id= SklOUpEYvB .

W. Liang, A. Keki´ c, J. von Kügelgen, S. Buchholz, M. Besserve, L. Gresele, and B. Schölkopf. Causal component analysis. In Advances in Neural Information Processing Systems , 2023.

P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and S. Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning , pages 13557-13603. PMLR, 2022.

C. Liu, L. Zhu, and M. Belkin. Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks. Applied and Computational Harmonic Analysis , 59:85116, 2022. ISSN 1063-5203. doi: https://doi.org/10.1016/j.acha.2021.12.009. URL https:// www.sciencedirect.com/science/article/pii/S106352032100110X . Special Issue on Harmonic Analysis and Machine Learning.

Y. Liu, Z. Zhang, D. Gong, M. Gong, B. Huang, A. v. d. Hengel, K. Zhang, and J. Q. Shi. Identifying weight-variant latent causal models. arXiv preprint arXiv:2208.14153 , 2022.

Y. Liu, A. Alahi, C. Russell, M. Horn, D. Zietlow, B. Schölkopf, and F. Locatello. Causal triplet: An open challenge for intervention-centric causal representation learning. In 2nd Conference on Causal Learning and Reasoning (CLeaR) , 2023. arXiv:2301.05169.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 4114-4124. PMLR, 2019.

I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https: //openreview.net/forum?id=Skq89Scxx .

M. Lotfollahi, A. K. Susmelj, C. De Donno, Y. Ji, I. L. Ibarra, F. A. Wolf, N. Yakubova, F. J. Theis, and D. Lopez-Paz. Compositional perturbation autoencoder for single-cell response modeling. BioRxiv , 2021.

Y. Lu and J. Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 2000f6325dfc4fc3201fc45ed01c7a5d-Abstract.html .

G. E. Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research , 2022.

H. Morioka and A. Hyvarinen. Connectivity-contrastive learning: Combining causal discovery and representation learning for multimodal data. In International Conference on Artificial Intelligence and Statistics , pages 3399-3426. PMLR, 2023.

J. Moser. On the volume elements on a manifold. Transactions of the American Mathematical Society , 120(2):286-294, 1965. ISSN 00029947. URL http://www.jstor.org/stable/ 1994022 .

A. Nejatbakhsh, F. Fumarola, S. Esteki, T. Toyoizumi, R. Kiani, and L. Mazzucato. Predicting perturbation effects from resting activity using functional causal flow. bioRxiv , pages 2020-11, 2020.

T. M. Norman, M. A. Horlbeck, J. M. Replogle, A. Y. Ge, A. Xu, M. Jost, L. A. Gilbert, and J. S. Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science , 365(6455):786-793, 2019.

OpenAI. GPT-4 technical report, 2023.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html .

J. Pearl. Causality . Cambridge university press, 2009.

J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 78(5):947-1012, 2016. ISSN 13697412, 14679868. URL http: //www.jstor.org/stable/44682904 .

J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

G. Rajendran, B. Kivva, M. Gao, and B. Aragam. Structure learning in polynomial time: Greedy algorithms, bregman information, and exponential families. Advances in Neural Information Processing Systems , 34:18660-18672, 2021.

G. Rajendran, P. Reizinger, W. Brendel, and P. K. Ravikumar. An interventional perspective on identifiability in gaussian lti systems with independent component analysis. In UAI 2023 Workshop on Causal inference for time series data , 2023.

A. G. Reisach, C. Seiler, and S. Weichwald. Beware of the simulated dag! causal discovery benchmarks may be easy to game. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 614, 2021, virtual , pages 27772-27784, 2021. URL https://proceedings.neurips.cc/ paper/2021/hash/e987eff4a7c7b7e580d659feb6f60c1a-Abstract.html .

A. G. Reisach, M. Tami, C. Seiler, A. Chambaz, and S. Weichwald. Simple sorting criteria help find the causal order in additive noise models. In Advances in Neural Information Processing Systems , 2023.

D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

J. Rissanen. Modeling by shortest data description. Autom. , 14(5):465-471, 1978. doi: 10.1016/ 0005-1098(78)90005-5. URL https://doi.org/10.1016/0005-1098(78)90005-5 .

G. Roeder, L. Metz, and D. Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning , pages 9030-9039. PMLR, 2021.

E. Rosenfeld, P. K. Ravikumar, and A. Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=BbNIbVPJ-42 .

E. Rosenfeld, P. Ravikumar, and A. Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856 , 2022.

D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(2):215-246, 2021.

B. Schölkopf and J. von Kügelgen. From statistical to causal learning. arXiv preprint arXiv:2204.00607 , 2022.

B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021. arXiv:2102.11107.

X. Shen, F. Liu, H. Dong, Q. Lian, Z. Chen, and T. Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research , 23:1-55, 2022.

P. Shinners. Pygame. http://pygame.org/ , 2011.

R. Silva, R. Scheines, C. Glymour, P. Spirtes, and D. M. Chickering. Learning the structure of linear latent variable models. Journal of Machine Learning Research , 7(2), 2006.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

P. Spirtes and C. Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review , 9(1):62-72, 1991.

P. Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search . MIT press, 2000.

C. Squires and C. Uhler. Causal structure learning: a combinatorial perspective. Foundations of Computational Mathematics , pages 1-35, 2022.

C. Squires, Y. Wang, and C. Uhler. Permutation-based causal structure learning with unknown intervention targets. In Conference on Uncertainty in Artificial Intelligence , pages 1039-1048. PMLR, 2020.

C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 32540-32560. PMLR, 2023. URL https://proceedings.mlr.press/v202/squires23a.html .

S. G. Stark, J. Ficek, F. Locatello, X. Bonilla, S. Chevrier, F. Singer, G. Rätsch, and K.-V. Lehmann. Scim: universal single-cell matching with unpaired feature sets. Bioinformatics , 36 (Supplement_2):i919-i927, 2020.

T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems , 33:3362-3373, 2020.

B. Varici, K. Shanmugam, P. Sattigeri, and A. Tajer. Scalable intervention target estimation in linear models. Advances in Neural Information Processing Systems , 34:1494-1505, 2021.

B. Varici, E. Acarturk, K. Shanmugam, A. Kumar, and A. Tajer. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

C. Villani. Optimal Transport: Old and New . Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008. ISBN 9783540710509. URL https://books.google. de/books?id=hV8o5R7_5tkC .

J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in Neural Information Processing Systems , 34, 2021.

J. von Kügelgen, M. Besserve, W. Liang, L. Gresele, A. Keki´ c, E. Bareinboim, D. M. Blei, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems , 2023.

Y. Wang, D. Blei, and J. P. Cunningham. Posterior collapse and latent variable nonidentifiability. Advances in Neural Information Processing Systems , 34:5443-5455, 2021.

S. Weichwald, S. W. Mogensen, T. E. Lee, D. Baumann, O. Kroemer, I. Guyon, S. Trimpe, J. Peters, and N. Pfister. Learning by doing: Controlling a dynamical system using causality, control, and reinforcement learning. In NeurIPS 2021 Competitions and Demonstrations Track , pages 246-258. PMLR, 2022.

M. Willetts and B. Paige. I don't need u : Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05238 , 2021.

F. Xie, R. Cai, B. Huang, C. Glymour, Z. Hao, and K. Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in neural information processing systems , 33:14891-14902, 2020.

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9593-9602, June 2021.

Y. Yu, J. Chen, T. Gao, and M. Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning , pages 7154-7163. PMLR, 2019.

J. Zhang, C. Squires, and C. Uhler. Matching a desired causal state via shift interventions. Advances in Neural Information Processing Systems , 34:19923-19934, 2021.

J. Zhang, C. Squires, K. Greenewald, A. Srivastava, K. Shanmugam, and C. Uhler. Identifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.06250 , 2023.

S. Zhao, J. Song, and S. Ermon. Infovae: Balancing learning and inference in variational autoencoders. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 , pages 5885-5892. AAAI Press, 2019. doi: 10.1609/AAAI.V33I01.33015885. URL https://doi.org/10.1609/aaai.v33i01. 33015885 .

X. Zheng, B. Aragam, P. K. Ravikumar, and E. P. Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems , 31, 2018.

Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: sparsity and beyond. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 6801fa3fd290229efc490ee0cf1c5687-Abstract-Conference.html .

R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021.

A Proof of Identifiability up to Linear Maps

In this section we give a full proof of Theorem 3. We try to make this essential self-contained. To make this easy to read, we first introduce and recall the required notation in Section A.1, then we prove two important key relations required for the proof in Section A.2. Finally, we prove first a simpler version of Theorem 3 with more assumptions in Section A.3 and then give the full proof of Theorem 3 in Section A.4.

A.1 General Notation

Let us introduce some notation for our identifiability proofs. For a full list of assumptions we refer to the main text, here we just recall the relevant notation concisely. We will assume that there are two representations that generate the same observations, i.e., we assume that there are two sets of Gaussian SCMs

where ϵ, ˜ ϵ ∼ N (0 , Id) and i = 0 corresponds to the observational distribution while i > 0 correspond to interventional settings where we intervene on a single node t i (or ˜ t i ) by adding a shift and changing the relation to its parents (without adding new parents). Moreover, there are two functions f , ˜ f such that X ( i ) D = f ( Z ( i ) ) and X ( i ) D = ˜ f ( ˜ Z ( i ) ) . It is convenient to define

so that Z ( i ) = ( B ( i ) ) -1 ϵ + µ ( i ) with ϵ ∼ N (0 , Id) . Note that all model information can be collected in (( B ( i ) , η ( i ) , t i ) , f ) .

As in the main text, it is convenient to use the shorthand B = B (0) , A = A (0) .

We denote the row t i of B ( i ) by r ( i ) , i.e.,

Note that for perfect interventions where we cut the relation to all parents of node t i we have r ( i ) = λ i e t i for some λ i = 0 . Similarly, for the observational distribution we refer to the row t i of B (0) by s ( i ) , i.e.,

We thus find

We also consider the covariance matrix Σ ( i ) and the precision matrix Θ ( i ) of Z ( i ) which are given by

Indeed,

Finally, the mean µ ( i ) of Z ( i ) given by

We also define similar quantities for ˜ Z . To simplify the notation, we will frequently use the shorthand v ⊗ w = v · w ⊤ ∈ R d × d for v, w ∈ R d .

̸

A.2 Two auxiliary lemmas

The key to our identifiability results is the relation shown in the following lemma.

Lemma 2. Assume that there are two latent variable models (( B ( i ) , η ( i ) , t i ) i ∈ I , f ) and (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ I , ˜ f ) as defined above such that f ( Z ( i ) ) D = ˜ f ( ˜ Z ( i ) ) . Then their latent dimension d agree and h = ˜ f -1 ◦ f exists and is a diffeomorphism, i.e., bijective and differentiable with differentiable inverse. Moreover, it satisfies the relation

for all z and i and some constant c ( i ) . If η ( i ) = ˜ η ( i ) = 0 this simplifies to

Proof. Note first that X (0) D = f ( Z (0) implies since f is an embedding by assumption that the dimension of the support of X (0) is the same as the latent dimension and this dimension is identifiable from the observations, e.g., by considering the tangent space. As Z (0) and ˜ Z (0) have full support and f and ˜ f are by assumption embeddings, the equality f ( Z (0) ) D = ˜ f ( ˜ Z (0) ) implies that the two image manifolds agree and h = ˜ f -1 ◦ f is a differentiable map. Moreover, we find that h ∗ Z ( i ) = ˜ Z ( i ) where h ∗ denotes the pushforward map. The variables Z ( i ) are Gaussian with distribution Z ( i ) ∼ N ( µ ( i ) , Σ ( i ) ) . This implies that its density p ( i ) can be written as

A similar relation holds for ˜ p ( i ) ( z ) , the density of ˜ Z ( i ) . Finally, we have the relations

which is a consequence of h ∗ Z ( i ) = ˜ Z ( i ) . Here J h denotes the Jacobian matrix of partial derivatives. Thus we conclude that

Calling this relation E i , we consider the difference E 0 -E i . Then the determinant term cancels, and we obtain for some constant c ( i )

Expanding the quadratic expression on the left-hand side we obtain

Finally we simplify the last term using (18)

Apply the same simplification on the right hand side. Finally, we collect the constant terms independent of z in c ( i ) , to obtain

which is (19) as desired.

A second key ingredient for our identifiability results is that the specific structure of the interventions that each target a single node which implies that the difference of precision matrices Θ ( i ) -Θ (0) has rank at most 2. This is also the key ingredient used (in a very different manner) in the proof of the main result in [84]. Here we highlight this simple result because we make use of it frequently.

Lemma 3. The precision matrices satisfy the relation

Proof. Using (17) we find

̸

Here we used the fact that all rows except row t i of B ( i ) and B (0) agree as the intervention targets a single node, i.e., ( B ( i ) ) ⊤ e k = ( B (0) ) ⊤ e k for k = t i .

A.3 Warm-up

To provide some intuition how identifiability arises, we first provide the proof for a simpler result with a much stronger assumption on the set of intervention targets and the type of interventions. But note importantly that we don't relax the assumption of non-linearity of f .

Theorem 4. Let f ( Z ( i ) ) D = ˜ f ( ˜ Z ( i ) ) for two sets of parameters (( B ( i ) , η ( i ) , t i ) i ∈ I , f ) and (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ I , ˜ f ) as in the above model. We assume that there are 2 n perfect pairwise dif-

̸

ferent interventions (i.e., Z ( i ) D = Z ( i ′ ) for i = i ′ ), without shifts (i.e., η ( i ) = ˜ η ( i ) = 0 ). Finally, we assume that there are 2 interventions for each node, and we know the intervention targets of the interventions, i.e., we assume that t i = ˜ t i for all i . Then, Z ( i ) and ˜ Z ( i ) agree up to scaling, i.e., Z ( i ) = D ˜ Z ( i ) for a diagonal matrix D ∈ Diag( n ) .

̸

Remark 4. Actually this result can be generalized to a setting where the latent SCM is non-parametric [92].

Proof. We consider a node k and two interventions i 1 and i 2 which target node k = t i 1 = t i 2 (by assumption those interventions exist and i 1 and i 2 agree for both representations). Since we assumed that the interventions are perfect, we have in our notation that

for two constants λ i 1 = λ i 2 > 0 and similarly for ˜ r ( i j ) . Indeed, assuming positivity is not a restriction since ϵ follows a symmetric distribution (this is also contained in the parametrisation ( D ( i ) ) 1 2 ) and λ i 1 = λ i 2 is necessary to ensure that the interventional distributions differ. Combining (20) from Lemma 2 with Lemma 3 we obtain the relation

̸

for j = 1 , 2 .

̸

Note that since k = t i 1 = t i 2 , we have s ( i 1 ) = s ( i 2 ) = B ⊤ e k by definition. Thus, we subtract the relation in the last display for i 1 from the relation for i 2 and find using (30),

̸

for all z . Since ˜ λ i 1 = ˜ λ i 2 > 0 we can divide and obtain

Since h is surjective, we have that h k is also surjective which implies that the range of the right hand side is [2( c ( i 1 ) -c ( i 2 ) ) / ( ˜ λ 2 i 1 -˜ λ 2 i 2 ) , ∞ ) . The range of the left hand side is depending on the sign of the prefactor and is either ( -∞ , 0] or [0 , ∞ ) . Since the ranges have to agree we conclude that c ( i 1 ) = c ( i 2 ) and

Since k was arbitrary, this ends the proof.

A.4 Proof of identifiability up to linear maps

The proof of our main result relies on two essentially geometric lemmas and a slight extension of Lemma 2 which we discuss now. The first lemma alone is sufficient to handle the case where interventions change the scaling matrix D , while the second is required to handle interventions that change B and the third allows us to consider pure shift interventions.

Lemma 4. Let g : R d → R be a continuous and surjective function, Q a quadratic form on R d and α ∈ R a constant such that

Then Q has rank 1 and g is linear, i.e., g ( z ) = v · z for some v ∈ R d .

̸

Proof. Assume that Q has the eigendecomposition Q = ∑ r i =1 λ i v i v ⊤ i where r is the rank of Q and λ i = 0 and v i has unit norm. Clearly g surjective implies r > 0 . Since g is surjective we find that the range of the left hand side is [0 , ∞ ) and therefore z · Qz ≥ -α . This implies that λ i ≥ 0 for all i and therefore, the range of the right hand side is [ α, ∞ ) which implies α = 0 . Now we fix any c > 0 and consider E c = { z ∈ R d | z · Qz = c } . It is easy to see that E c is the product of an ellipsoid and R d -r and thus is connected if r ≥ 2 . We find by (35) that g ( z ) = ± √ c for z ∈ E c and if g ( z ) ∈ {- √ c, √ c } then z ∈ E c . By continuity of g and since E c is connected this implies that g ( z ) = √ c for all z ∈ E c or g ( z ) = - √ c for all z ∈ E c contradicting the surjectivity of g . We conclude that r = 1 , i.e., Q = λ 1 v 1 v ⊤ 1 = ˜ v 1 ˜ v ⊤ 1 with ˜ v 1 = √ λ 1 v 1 . Thus we find that

Since g is continuous, this implies that g ( z ) = ± ˜ v 1 · z or g ( z ) = ±| ˜ v 1 · z | . As only the first functions are surjective we conclude that g ( z ) = ± ˜ v 1 · z and the claim follows with w = ± ˜ v 1 .

To handle shift interventions we need a slightly stronger version of the lemma above which contains an additional linear term.

Corollary 1. Let g : R d → R be a continuous and surjective function, Q a quadratic form on R d , w ∈ R d a vector, and α ∈ R a constant such that

Then Q has rank 1 and g is affine, i.e., g ( z ) = v · z + c for some v ∈ R d and c ∈ R .

̸

Proof. Assume as before that Q has the eigendecomposition Q = ∑ r i =1 λ i v i v ⊤ i where r is the rank of Q and λ i = 0 and | v i | = 1 . First, we show that w ∈ ⟨ v 1 , . . . , v r ⟩ where ⟨ . ⟩ denotes the span. Indeed, suppose that this is not the case. Then we could find z 0 such that z 0 · w < 0 and z 0 v i = 0 for 1 ≤ i ≤ r , and therefore z 0 · Qz 0 = 0 . Plugging z = λz 0 for λ → ∞ in (37) the right-hand side tends to -∞ while the left-hand side is non-negative. This is a contradiction and therefore w ∈ ⟨ v 1 , . . . , v r ⟩ holds. This implies that there is w ′ such that Qw ′ = w/ 2 and thus

Then we consider ˜ g ( ˜ z ) = g ( ˜ z -w ′ ) , ˜ α = α -w ′ · Qw ′ which satisfy

Using Lemma 4 we conclude that Q has rank 1 and ˜ g ( ˜ z ) = v · z for some v . But then g ( z ) = ˜ g ( z + w ′ ) = z · v + w ′ · v which concludes the proof.

The second lemma is similar, but slightly simpler.

Lemma 5. Assume that there is a quadratic form Q , a vector 0 = w ∈ R d , w ′ ∈ R d , α ∈ R and a continuous function g : R d → R such that

̸

Then g ( z ) = v · z + c for some v ∈ R d , c ∈ R , i.e., g is an affine function.

Proof. By rescaling we can assume that w has unit norm. Plugging in z = 0 we find that α = 0 . Denote by Π w the orthogonal projection on w , i.e., Π w v = ( w · v ) w and by Π ⊥ w the projection on the orthogonal complement of w , i.e., z = Π w z +Π ⊥ w z for all z ∈ R d . Then we find for all z

By scaling z we find that both terms on the right hand side vanish and thus for all z

This implies (using also the symmetry of Q )

Now we use Π w z = ( w · Π w z ) w and find

This implies

̸

for all z such that Π w z = 0 . By continuity we conclude that this holds for all z , i.e., g is affine.

Again we need a slight generalization of this lemma.

Corollary 2. Assume that there is a quadratic form Q , a vector 0 = w ∈ R d , w ′ ∈ R d , α, β ∈ R and a continuous function g : R d → R such that

̸

Then g ( z ) = v · z + c for some v ∈ R d , c ∈ R , i.e., g is an affine function.

Proof. Define ˜ g ( ˜ z ) = g ( ˜ z -βw | w | -2 ) . Then

for some ˜ w and ˜ α . So, we conclude from Lemma 5 that ˜ g is affine and thus the same is true for g .

We now discuss a small extension of Lemma 2 by showing that we can complete the squares in (19) which then allows us to obtain a relation that is very similar to (20).

̸

Lemma 6. Assume the general setup as introduced above and also Θ ( i ) = Θ . Then there is a vector µ ′ ( i ) and a constant c such that

Proof. Using the symmetry of the precision matrices and expanding all the terms, we can express the difference of the left-hand side and the right-hand side of (51) as

Hence, we can set c at the end such the constant terms cancel and all we have to show is that there exists µ ′ ( i ) such that

This is clearly not true for arbitrary µ ( i ) because Θ ( i ) -Θ has only rank 2. However, such a µ ′ ( i ) does exist for µ ( i ) = η ( i ) ( B ( i ) ) -1 e t i (see equation (18)). Indeed, we can simplify using (17) and Lemma 3

Now there are two cases. When r ( i ) and s ( i ) are collinear (but not identical, by assumption) we can choose µ ′ ( i ) ) = λr ( i ) for a suitable λ . Otherwise, we can pick any vector µ ′ ( i ) such that µ ′ ( i ) s ( i ) = 0 , µ ′ ( i ) r ( i ) = η ( i ) . This is always possible for non-collinear vectors (project r ( i ) on the orthogonal complement of s ( i ) ).

Let us now first discuss a rough sketch of the proof of our main result. This allows us to present the main ideas without discussing all the technical details required for the full proof of the result. We restate the theorem for convenience.

Theorem 3. Assume that X ( i ) is generated according to a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) such that the Assumptions 1-4 hold. Then we have identifiability up to linear transformations, i.e., if (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) , ˜ f ) generates the same observed distributions ˜ f ( ˜ Z ( i ) ) D = X ( i ) , then their latent dimensions d agree and there is an invertible linear map T such that

Proof Sketch. For simplicity of the sketch we ignore shift interventions here. We assume that we have two sets of parameters (( B ( i ) , t i ) i ∈ ¯ I , f ) , ( ˜ B ( i ) , ˜ t i ) i ∈ ¯ I ) , ˜ f ) generating the same observations. We can relabel the interventions i and the variables ˜ Z k such that ˜ t i = i and such that 1 , 2 , . . . is a valid causal order of the graph. The general idea is to show by induction that ( h 1 ( z ) , . . . , h k ( z )) is a linear function of z . Let us sketch how this is achieved. The key ingredients here are Lemma 2 and Lemma 3. Applying (20) from Lemma 2 in combination with Lemma 3 we obtain

By our assumption that the variables ˜ Z ( i ) follow the causal order of the underlying graph, only the entries ˜ r ( k +1) r for r ≤ k +1 are non-zero and similarly for ˜ s ( k +1) . Therefore, the right-hand side is a quadratic form of ( h 1 ( z ) , . . . , h k +1 ( z )) . Now we apply our induction hypothesis which states that ( h 1 ( z ) , . . . , h k ( z )) is a linear function of z . Then we find expanding the quadratic form on the right-hand side that there is a matrix Q , a vector v , a constant δ (which we here assume to be non-zero) and another quadratic form Q ′ such that

Now we can apply Corollary 1 to conclude that h k +1 ( z ) is an affine function of z . Adding the shifts and considering also the case δ = 0 adds several technical difficulties, which we handle in the full proof.

After all those preparations and presenting the proof sketch, we will finally prove our main theorem in full generality.

Full proof of Theorem 3. As the proof has to consider different cases and is a bit technical we structure it in a series of steps.

Labeling assumptions We can assume without loss of generality (by relabeling variables and interventions) that the variables ˜ Z i are ordered such that their natural order i = 1 , 2 , . . . is a valid topological order of the underlying DAG and that intervention 1 ≤ i ≤ n targets node Z i , i.e., ˜ t i = i . We make no assumption on the ordering of the Z i or targets of t i which are not necessarily assumed to be pairwise different. Recall that we defined (and showed existence of) h = ˜ f -1 f in Lemma 2 (including the fact that their latent dimensions agree).

Induction claim We now prove by induction that h k ( z ) is linear for k ≥ 0 . The base case k = 0 is trivially true. For the induction step, assume that this is the case for j ≤ k , i.e., h j ( z ) = m j · z for some m j ∈ R d . We define M k = ( m 1 , . . . , m k ) ⊤ so that we can write concisely

Note that since h is surjective M k has full rank. To prove the induction step, we need to show that there exists m k +1 ∈ R d such that h k +1 ( z ) = m k +1 · z .

̸

Precision matrix decomposition For the DAG ˜ G underlying ˜ Z i , let PA i denote the indices of the set of parents of Z i and let PA i = { i } ∪ PA i . Recall the notation ˜ s ( i ) = ˜ B ⊤ e i (using ˜ t i = i ) and ˜ r ( i ) = ( ˜ B ( i ) ) ⊤ e i . Now ˜ s ( i ) j = 0 only holds if j ∈ PA i and we ordered the variables such that PA i ⊂ [ i ] . In particular, ˜ s ( i ) j = 0 and ˜ r ( i ) j = 0 for j > i (interventions do not create new parents). By Lemma 3 we have

We find that

if r > k +1 or s > k +1 . In particular, there exists a symmetric A ∈ R ( k +1) × ( k +1) such that

Now we decompose A as

for some A ′ ∈ R k × k , b ∈ R k and δ ∈ R . Using the induction hypothesis we find

We use the notation v : r ∈ R r for the vector ( v 1 , . . . , v r ) ⊤ . Then we can write

Now we have to consider different cases.

̸

Case δ = 0 We assume first that δ = 0 . In this case we find using the last two displays

̸

Next we set

Looking carefully at (65) we find that there is a symmetric matrix ˜ Q , a vector ˜ w , and a constant ˜ c such that

We claim that g is continuous and surjective. Continuity follows directly from continuity of h . To show that g is surjective we can ignore the constant shift, and we note that δ -1 v ⊤ M k z = δ -1 ∑ k r =1 v r h r ( z ) and thus surjectivity of h implies that g is surjective (for every c pick z such that h k +1 ( z ) = c and h r ( z ) = 0 for r ≤ k ).

Using (19) from Lemma 2 we conclude that

for all z and some symmetric Q ∈ R d × d , w ∈ R d . Corollary 1 then implies that

for some v ∈ R d , c ∈ R . Thus h k +1 is an affine function, i.e., h k +1 ( z ) = m k +1 z + α . But

and therefore h k +1 is linear.

̸

Case δ = 0 , b = 0 Now we consider the case that 0 = δ . Using (63), (64), and (19) we find similarly to above

Collecting all terms we find that h k +1 satisfies a relation of the type

̸

for suitable Q , β , w , w ′ , and α . Note in particular that w ⊤ = 2 b ⊤ M k = 0 because M k has full rank. If w = 0 then Corollary 2 implies that h k ( z ) is affine and, as argued above, we conclude that h k is linear.

̸

Case δ = 0 , b = 0 It remains to address the case b = 0 and δ = 0 . Going back to the definition of b and δ (see (61) and (62)) we find

Now δ = 0 implies ˜ s ( k +1) k +1 = ˜ r ( k +1) k +1 (because they are both positive). But then we conclude ˜ r ( k +1) = ˜ s ( k +1) from b = 0 and ˜ r ( k +1) r = ˜ s ( k +1) r = 0 for r > k +1 . This implies

̸

That is, this implies that intervention k + 1 is a pure shift intervention, i.e., B = B ( k +1) and η ( k +1) = 0 (otherwise we do not actually intervene). In this case, we conclude from Lemma 2 and Lemma 6 (if Θ ( k +1) = Θ )

̸

But then η ( k +1) ˜ r ( k +1) ∇ h ( µ ′ ( k +1) ) = 0 , this implies that ∇ h is not invertible which is a contradiction to h being a diffeomorphism. Thus, we conclude that also Θ ( k +1) = Θ , i.e., intervention k +1 is also a pure shift intervention for the Z variables. But then we find

So we finally conclude that also in this case h k +1 is a linear function, this ends the proof.

B From Linear Identifiability to Full Identifiability

In this section, we provide the proof of Theorem 1 which is a combination of our linear identifiability result Theorem 3 proved in the previous section and the main result of [84] which shows identifiability for linear mixing f . However, we need to carefully address their normalization choices and in addition consider shifts.

We emphasize that the full identifiability results could also be derived from our proof of Theorem 3 by carefully considering ranks of quadratic forms and showing that we can inductively identify source nodes. We do not add these arguments as our reasoning is already quite complex and because for linear mixing functions this is not substantially different from the original proof in [84].

For the convenience of the reader, we will now restate the main results of [84]. For our work it is convenient to slightly deviate from their conventions, i.e., we define the causal order of a graph by i ≺ j iff i ∈ an( j ) , in particular i → j implies i ≺ j .

We rephrase their results using our notation. Let S ( G ) denote the set of permutations of the vertices of G that preserve the causal ordering, i.e., all permutations ρ such that ρ ( i ) < ρ ( j ) if i ≺ j . They consider linear mixing functions as specified next.

Assumption 5. Assume f ( z ) = Lz for some matrix L ∈ R d ′ × d with trivial kernel and pseudoinverse H = L + such that the largest absolute value of each row of H is one and the leftmost is positive.

Then the following result holds.

Theorem 5 (Linear setting, Theorems 1 and 2 in [84]) . Suppose latent variables Z ( i ) are generated following Assumptions 2 and 3 where η ( i ) = 0 (i.e., no shift) with DAG G and all A ( i ) are lower triangular. Suppose in addition that Assumption 4 holds and the mixing function satisfies Assumption 5. Then we can identify intervention targets and the partial order ≺ G of the underlying DAG G up to permutation of the node labels. If we in addition assume that all interventions are perfect the problem is identifiable in the following sense. For every representation ( ˜ B ( i ) , ˜ H ) such that all ˜ B ( i ) are lower triangular and ˜ L ( ˜ Z ( i ) ) D = X ( i ) where ˜ L = ˜ H + there is a permutation σ ∈ S ( G ) such that

Remark 5. A few remarks are in order.

Compared to their statement in Theorem 2 we replaced P σ by P ⊤ σ because we used the transposed convention for the permutation matrices.

Note that their result per se doesn't mention identifiability up to scaling but that's because part of their assumptions involve normalizing f . Since we don't normalize the linear map f , we add scaling in the theorem statement.

For the case of imperfect interventions, their result talks about identifiability of the transitive closure G of the graph G , but this is the same as identifiability of the partial order ≺ G that we state here.

We assume that interventions are not only acting on the noise but do change the corresponding row of A . Note that this is equivalent to their genericity assumption which states that

̸

are linearly independent. Indeed, this holds iff A (0) t i · = A ( i ) t i · (by acyclicity A ( i ) t i t i = 0 ).

Let us provide a brief intuition why this result holds. The key ingredient is the same as in our proof of linear identifiability above, namely, Lemma 3 which reads

This expression has rank 2 if t i is not a source node in the causal graph and the intervention is not a pure noise intervention because s ( i ) and r ( i ) are linearly independent as explained in the remark above. On the other hand, this expression has rank one for source nodes because s ( i ) ∝ e t i in this case and the same is true for r ( i ) . This allows us to identify interventions on source nodes. Moreover, Θ ( i ) -Θ ( k ) has rank 2 if intervention i and k target different source nodes and rank 1 if they target the same source node. Thus we can also identify the set of interventions targeting the same source node. One can complete the proof by an induction argument based on removing source nodes from the graph and studying the effect on the precision matrix of the latent variables.

We are now ready to prove our main theorems.

Theorem 1. Suppose we are given distributions X ( i ) generated using a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) such that Assumptions 1-4 hold and such that all interventions i are perfect. Then the model is identifiable up to permutation and scaling, i.e., for any model (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ ¯ I , ˜ f ) that generates the same data, the latent dimension d agrees and there is a permutation ω ∈ S d (and associated permutation matrix P ω ) and an invertible pointwise scaling matrix Λ ∈ Diag( d ) such that

This in particular implies that

and we can identify the causal graph G up to permutation of the labels.

Proof of Theorem 1. Suppose there are two representations (( B ( i ) , η ( i ) , t i ) i ∈ I , f ) and (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ I , ˜ f ) of the distributions X ( i ) , i ≥ 0 . As explained in the proof of Theorem 3 we can assume by relabeling the nodes that the matrices ˜ B ( i ) are lower triangular and the corresponding DAG ˜ G has the property that for every edge i → j we have i < j . We now apply Theorem 3 and find that there is an invertible linear map T such that ˜ f = f ◦ T and T -1 Z ( i ) D = ˜ Z ( i ) . One minor difference to the setting in [84] is that we in addition consider shift interventions. They are simpler to analyze than a change of variance, but we can avoid additional arguments with the following trick. If Σ ( i ) = Σ (0) for some i we replace Z ( i ) = N ( µ ( i ) , Σ ( i ) ) by Z ′ ( i ) = N (0 , Σ ( i ) + µ ( i ) ( µ ( i ) ) ⊤ ) , otherwise we set Z ′ ( i ) = N (0 , Σ ( i ) ) and similarly for ˜ Z . Then T -1 ( Z ( i ) ) D = ˜ Z ( i ) implies T -1 ( Z ′ ( i ) ) D = ˜ Z ′ ( i ) . Using (13) and (17) it can be shown that this distribution corresponds to a change of variance of the noise, i.e., a change of D ( i ) t i t i . This can also be seen directly from the relation Z ′ ( i ) D = ( B ( i ) ) -1 ( ϵ + η ( i ) ϵ ′ e t ( i ) ) , where ϵ ′ ∼ N (0 , 1) and

independent of ϵ . We can now apply Theorem 5 to the primed distributions. We first find a unique invertible Λ ∈ Diag( d ) such that Λ T has largest absolute value entry 1 in every row (and in case of ties the leftmost entry is 1). We can also find a permutation ρ ∈ S d such that ¯ B ( i ) = P ρ B ( i ) Λ -1 P ⊤ ρ is lower triangular for all i (note the distinction between ¯ B ( i ) and ˜ B ( i ) ). Indeed, this is possible since the underlying DAG G is acyclic by assumption, interventions do not add edges, and Λ is diagonal. Then we find

Here the last step follows from the fact that T -1 Z ( i ) D = ˜ Z ( i ) which implies that the centered distributions agree.

Now we can apply Theorem 5 (with B replaced by ˜ B and identity mixing) and find that for the alternative representation in terms of ¯ B

for some ρ ∈ S ( ˜ G ) because by assumption ¯ B ( i ) and ˜ B are lower triangular. This implies that

To summarize, there is a permutation ω = ( ρ ◦ σ ) -1 (note that P σ P ρ = P ρ ◦ σ ) and a diagonal matrix Λ such that

We also find the relation ω ( t i ) = ˜ t i from here because P ω e i = e ω -1 ( i ) . Equating the means of T -1 Z ( i ) and ˜ Z ( i ) we find

Multiplying by ˜ B ( i ) we find

and we conclude that ˜ η ( i ) = η ( i ) .

Theorem 2. Suppose we are given the distributions X ( i ) generated using a model (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) with causal graph G such that the Assumptions 1-4 hold and none of the interventions is a pure noise intervention. Then for any other model (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ ¯ I , ˜ f ) with causal graph ˜ G generating the same observations the latent dimension d agrees and there is a permutation ω ∈ S d such that ˜ t i = ω ( t i ) and i ≺ ˜ G j iff ω ( i ) ≺ G ω ( j ) , i.e., ≺ G can be identified up to a permutation of the labels.

Proof of Theorem 2. Suppose there are two representations (( B ( i ) , η ( i ) , t i ) i ∈ I , f ) and (( ˜ B ( i ) , ˜ η ( i ) , ˜ t i ) i ∈ I , ˜ f ) of the observational distribution. As shown in Theorem 3 there is a linear map such that T -1 Z ( i ) = ˜ Z ( i ) . Then the same is true for the transformed variables as explained above. Viewing ˜ Z ( i ) as new observations obtained through a linear T mixing we conclude from Theorem 5 that ≺ G and the intervention targets are identifiable up to relabeling.

C A comparison to related works

In this section, we will discuss the relation of our results to the most relevant prior works and put them into context.

[84] This work considers linear mixing functions f , and we directly generalize their work to nonlinear f . In particular, their techniques are linear-algebraic and do not generalize to non-linear f (see more technical details in proof intuition in Section 4); we handle this using a mix of statistical and geometric techniques. Notably, their finding that the linearly mixed latent variables are recoverable after observing one intervention per latent node is closely related to an earlier result in domain generalization, which showed that a number of environments equal to the number of 'non-causal' latent dimensions can ensure recovery of the remaining latents [71], and later work showing that this requirement can be reduced further under linear mixing with additional modeling structure [11, 72]. This connection is not coincidental-that analysis was for invariant prediction with latent variables, which was originally presented as a method of causal inference using distributions resulting from unique interventions [62]. Later this approach was applied for the purpose of robust prediction in nonlinear settings [25], but the identifiability of the latent causal structure under general nonlinear mixing via interventional distributions remained an open question until this work.

[88] This work also considers linear mixing f , but allow for non-linearity in the latent variables. They use score functions to learn the model and they raise as an open question the setting of non-linear mixing, which we study in this work. While the models are not directly comparable, our model is more akin to real-world settings since it's not likely that high-level latent variables are linearly related to the observational distribution, e.g. pixels in an image are not linearly related to high-level concepts. Moreover, Gaussian priors and deep neural networks are extensively used in practice, and our theory as well as experimental methodology apply to them.

[48] While they do not discuss interventions, this work considers the setting of multi-environment CRL with Gaussian priors and their results can be applied to interventional learning. When applied to interventional data as in our setting (as also shown in [84, Appendix D]), they require additional restrictive assumptions such as d = d ′ , a bijective mixing f and e ≤ d where e is the number of edges in the underlying causal graph whereas we make no such restrictions on d, f and e (therefore, the maximum value of e can be as large as ≈ d 2 / 2 in our setting). And when these assumptions are satisfied, they require 2 d interventions whereas we show that d interventions are sufficient as well as necessary.

[2] This work also considers the setup of interventional causal representation learning. However, their setting is different in various ways and we now outline the key differences.

Their main results assume that the mixing f is an injective polynomial of finite degree p (see Assumption 2 in their paper). The injectivity requires their coefficient matrix to be full rank, which in turn implies d ′ ≥ ∑ p r =0 ( r + d -1 d -1 ) = ( p + d d ) (see the discussion below Assumption 2 in their work). This means that we cannot set the degree of the decoder polynomial to be arbitrarily large for fixed output dimension d ′ (which is required for universal approximation). In contrast, we only require d ′ ≥ d . For a general discussion of the relation between approximability and identifiability we refer to Appendix E.

All their results for general nonlinear mixing functions rely on deterministic do-interventions which is a type of hard intervention that assigns a constant value to the target. However, we focus on randomized soft interventions (and also allow shift interventions which have found applications [73, 58]), which as also pointed out by [84, 87] is less restricted. Moreover, they require multiple interventions for each latent variable (as they use an ϵ -net argument) to show approximate identifiability, while the structural assumptions on the causal model allow us to show full identifiability with just one intervention for each latent variable. Thus, their setting is not directly comparable to ours and neither is strictly more general than the other. In particular, their proof techniques and experimental methodology don't translate to our setting, and we use completely different ideas for our proofs and experiments.

D Counterexamples and Discussions

In this section, we first present several counterexamples to relaxed versions of our main results in Section D.1 and then provide the missing proofs in Section D.2.

D.1 Main counterexamples

In this section, we will discuss the optimality and limitations of various assumptions of our main results. In particular, we consider the number of interventions, the distribution of the latent variables, and the types of interventions separately.

Number of interventions Our main results all require d interventions, i.e., one intervention per node. This is necessary even in the simpler setting of linear f as was shown in [84, Proposition 4], directly implying it is also necessary for the more general class of non-linear f that we handle here. Therefore, the number of interventions in our main theorems 1, 2 is both necessary and sufficient. Going a step further, it is natural to ask whether Theorem 3 remains true for less than d interventions. However, as we show next, d -2 interventions are not sufficient.

Fact 1. Suppose we are given distributions X ( i ) generated by (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) satisfying Assumption 1-3. If the number of interventions satisfies | I | ≤ d -2 it is not possible to identify f up to linear maps. Consider, e.g., d = 2 and Z = N (0 , Id) and no interventions. Then h ( Z ) D = Z for (nonlinear) radius dependent rotations as defined in [29, 9].

Let us next consider the case with a total of d environments which corresponds to d -1 interventional distributions plus one observational distribution. We can provide a non-identifiability result for a general class of heterogeneous latent variable models that satisfy an algebraic property.

Lemma 7. Assume that d = 2 and Z (0) ∼ N (0 , Σ (0) ) , Z (1) ∼ N (0 , Σ (1) ) and Σ (1) ≻ Σ (0) or Σ (1) ≺ Σ (0) in Löewner order (i.e., the difference is positive definite). Then there is a non-linear map h such that h ( Z ( i ) ) D = Z ( i ) for i = 0 , 1 .

We provide a proof of this lemma in Appendix D.2. Note that this non-identifiability lemma requires the assumption on Σ ( i ) . In Lemma 3 in Appendix A we have seen that for the interventions considered in this work the relation Σ ( i ) > Σ (0) or Σ ( i ) < Σ (0) generally does not hold. This is some weak indication that for Theorem 3, d -1 interventions might be sufficient. We leave this for future work.

As an additional note, in the special case when f is a permutation matrix, we are in the setting of traditional causal discovery and here, d -1 interventions are necessary and sufficient [10, 18, 84].

Type of intervention Even when we restrict ourselves to only linear mixing functions as considered as in [84], we need perfect interventions in Theorem 1. Otherwise, only the weaker identifiability guarantees in Theorem 2 can be obtained. This is shown in [84]for linear f . Therefore, in the more general setting of nonlinear f , we cannot hope to obtain Theorem 1 with imperfect interventions, showing that this assumption is needed.

Next, we clarify that for the identifiability result in Theorem 2 , the condition that the interventions are not pure noise interventions is also necessary.

Fact 2. Recall that we defined B ( i ) = ( D ( i ) ) -1 2 (Id -A ( i ) ) . Suppose X ( i ) is generated by (( B ( i ) , η ( i ) , t i ) i ∈ ¯ I , f ) satisfying Assumptions 1-4 where all interventions are pure noise interventions. By definition, this implies A ( i ) = A for all i . Consider any matrix ˜ A such that the corresponding DAG is acyclic. Then (( ˜ B ( i ) , η ( i ) , t i ) i ∈ ¯ I , ˜ f ) with ˜ B ( i ) = ( D ( i ) ) -1 2 (Id -˜ A ) for all i and

˜ ˜ ˜ ˜ generates the same distributions X ( i ) . Indeed,

This implies that any underlying causal graph could generate the observations. In other words, for pure noise interventions we can identify the scale of the noise variables, but we obtain no intervention

about the causal variables and structure (exactly because the interventions do not target the actual causal variables but the noise variables).

Finally, we remark that in contrast to the case with linear mixing functions, we need to exclude non-stochastic hard interventions for linear identifiability.

Lemma 8. Consider d = 2 and Z (0) = N (0 , Id) and the non-stochastic hard interventions do( Z 1 = 0) and do( Z 2 = 0) resulting in the degenerate Gaussian distributions Z (1) = N (0 , e 2 ⊗ e 2 ) and Z (2) = N (0 , e 1 ⊗ e 1 ) . Then there is a nonlinear h such that h ( Z ( i ) ) D = Z ( i ) for all i .

Intuitively, to show this, we choose h to be identity close to the supports { Z 1 = 0 } and { Z 2 = 0 } but some nonlinear measure preserving deformation in the four quadrants. The full argument can be found in Appendix D.2. A similar counterexample was discussed in [2]. However, here we in addition constrain the distribution of the latent variables, which adds a bit of complexity.

Distribution of the Noise Variables Our key additional assumption compared to the setting considered in [84, 2, 88] is that we assume that the latent variables are Gaussian which allows us to put only very mild assumptions on the mixing function f . However, we do this in a manner so that we still preserve universal representation guarantees.

[88] consider arbitrary noise distributions but restrict to linear mixing functions. We believe that our result can be generalized to more general distributions of the noise variables, e.g., exponential families, but those generalizations will require different proof strategies, which we leave for future work. However, the result will not be true for arbitrary noise distributions, as the following lemma shows.

A proof of this lemma can be found in Appendix D.2. The result shows that even with d perfect interventions and linear SCMs it is not possible to identify the underlying DAG (empty graph for Z and ˜ Z 1 → ˜ Z 2 ) and we also cannot identify f up to linear maps. While our example relies on distributions with bounded support, we conjecture that full support is not sufficient for identifiability.

Structural Assumptions A final assumption used in our results is the restriction to linear SCMs. This is used for most works on causal representation learning (see Section 2), even when they restrict to linear mixing functions. Although this may potentially be a nontrivial restriction, the advantage is that the simplicity of the latent space will enable us to meaningfully probe the latent variables, intervene and reason about them. Nevertheless, it's an interesting direction to study to what generality this can be relaxed. Indeed, even for the arguably simpler problem of causal structure learning (i.e., when Z ( i ) is observed) there are many open questions when moving beyond additive noise models. We leave this for future work.

D.2 Technical constructions

In this section, we provide the proofs for the counterexamples in Section D.1. For the first two counterexamples we construct functions h such that h ( Z ( i ) ) D = Z ( i ) by setting h = Φ t where Φ t is defined as the flow of a suitable vectorfield X , i.e.,

This is a common technique in differential geometry and was introduced to construct counterexamples to identifiability ICA with volume preserving functions in [9, 8]. To find a suitable vectorfield X we rely on the fact that the density p t of P t = (Φ t ) ∗ P satisfies the continuity equation

In particular P t = P holds iff

Therefore it is sufficient to find a vectorfield X such that Div( p ( i ) X ) = 0 for all environments i to construct a suitable h = Φ 1 .

Lemma 7. Assume that d = 2 and Z (0) ∼ N (0 , Σ (0) ) , Z (1) ∼ N (0 , Σ (1) ) and Σ (1) ≻ Σ (0) or Σ (1) ≺ Σ (0) in Löewner order (i.e., the difference is positive definite). Then there is a non-linear map h such that h ( Z ( i ) ) D = Z ( i ) for i = 0 , 1 .

Proof of Lemma 7. Let us first discuss the high-level idea of the proof. The general strategy is to construct a vectorfield X whose flow preserves Z ( i ) for i = 0 , 1 . This holds if and only if the densities p 0 , p 1 of Z (0) , Z (1) satisfy

Using Div( fX ) = ∇ f · X + f Div X we find

Weconclude that any X satisfying (97) must be orthogonal to ∇ (ln p 0 ( z ) -ln p 1 ( z )) and thus parallel to the level lines of ln p 0 ( z ) -ln p 1 ( z ) . This already fixes the direction of X and the magnitude can be inferred from (97). To simplify the calculations we can first linearly transform the data so that the directions of X , i.e., equivalently the level lines of ln p 0 ( z ) -ln p 1 ( z ) have a simple form. We assume that Σ 0 ≺ Σ 1 . Clearly, it is sufficient to show the result for any linear transformation of the Gaussian distributions Z ( i ) . Note that generally for G ∼ N ( µ, Σ) we have AG ∼ N ( Aµ,A Σ A ⊤ ) Applying Σ -1 2 0 we can reduce the problem to Id ≺ Σ ′ 1 = Σ -1 2 0 Σ 1 Σ -1 2 0 . Diagonalizing Σ ′ 1 by U ∈ SO( d ) it is sufficient to consider Id ≺ Λ = U Σ ′ 1 U ⊤ where Λ = Diag( λ 1 , λ 2 ) . Finally, we can rescale z 1 and z 2 such that the resulting covariance matrices Λ 0 = Diag( λ 0 1 , λ 0 2 ) , Λ 1 = Diag( λ 1 1 , λ 1 2 ) satisfy

Thus, it is sufficient to show the claim for such covariances Λ 0 , Λ 1 and the result for arbitrary covariances follows by applying suitable linear transformation.

For such covariances we find for some constant c

The level lines are circles. Thus, we consider the vector field

We see directly that Div X 0 = 0 . We consider the Ansatz X ( z ) = f ( | z | ) X 0 ( z ) /p 0 ( z ) . We now fix the radial function f : R + → R such that it has compact support away from 0. We find using Div X 0 = 0

Note, that close to 0 where | z | is not differentiable the vector field vanishes by our construction of f . We find similarly using (100)

Z

(1)

Z

(0)

Z

(2)

R

2

R

1

˜

Z

(0)

˜

Z

(1)

˜

Z

(2)

Q

2

Q

1

h

Finally we remark that the flow Φ t of the vectorfield X generates a family of functions h as desired, i.e., for all t we have (Φ t ) ∗ Z ( i ) = Z ( i ) and Φ t is clearly nonlinear as it is the identity close to 0 but not globally.

The proof of the next lemma is similar but simpler.

Lemma 8. Consider d = 2 and Z (0) = N (0 , Id) and the non-stochastic hard interventions do( Z 1 = 0) and do( Z 2 = 0) resulting in the degenerate Gaussian distributions Z (1) = N (0 , e 2 ⊗ e 2 ) and Z (2) = N (0 , e 1 ⊗ e 1 ) . Then there is a nonlinear h such that h ( Z ( i ) ) D = Z ( i ) for all i .

Proof of Lemma 8. Pick any smooth divergence free vectorfield X 0 with compact support in [ ϵ, ∞ ) 2 for some ϵ > 0 . Then the vector field X = X 0 /p 0 satisfies

Thus the flow Φ t preserves the distribution of Z (0) . But it also preserved the interventional distributions Z ( i ) for i = 1 , 2 because X ( z ) = 0 and thus Φ t ( z ) = z for z close to the supports of Z ( i ) .

Finally we prove Lemma 9 that shows that we cannot completely drop the assumption on the distribution of the noise variables ϵ .

Lemma 9. Consider Z (0) = ( ϵ 1 , ϵ 2 ) ⊤ , Z (1) = ( ϵ ′ 1 , ϵ 2 ) , and Z (2) = ( ϵ 1 , ϵ ′ 2 ) where ϵ 1 , ϵ 2 ∼ U ([ -1 , 1]) and ϵ ′ 1 , ϵ ′ 2 ∼ U ([ -3 , 3]) . Also define ˜ Z (0) = ( ϵ 1 , ϵ 1 + ϵ 2 ) , ˜ Z (1) = ( ϵ ′ 1 , ϵ ′ 1 + ϵ 2 ) , and ˜ Z (2) = ( ϵ 1 , ϵ ′ 2 ) . Then there is h such that

Proof of Lemma 9. Asketch of the setting can be found in Figure 4. We define h 0 ( z ) = ( z 1 , z 1 + z 2 ) . If h ( z ) = h 0 ( z ) for -1 ≤ z 2 ≤ 1 we find h ( Z ( i ) ) D = ˜ Z ( i ) for i = 0 and i = 1 . It remains to modify h 0 such that h preserves the uniform measure on [ -1 , 1] × [ -3 , 3] . We do this in two steps, first we

construct a modification ˜ h of the map h 0 such that [ -1 , 1] × [ -3 , 3] is mapped bijectively to itself but ˜ h = h 0 for | z 2 | ≤ 1 (so that ˜ h ( Z ( i ) ) D = ˜ Z ( i ) for i = 0 and i = 1 holds) and then we apply a general result to make the map measure preserving. We start with the first step. Let ψ : R → [0 , 1] be differentiable such that ψ ( t ) = 1 for -1 ≤ z ≤ 1 and ψ ( t ) = 0 for -5 / 2 ≤ z ≤ 5 / 2 with | ψ ′ ( t ) | < 1 . Then

is injective on [ -1 , 1] × [ -3 , 3] (note that the second coordinate is increasing in z 2 ) and maps [ -1 , 1] × [ -3 , 3] bijectively to itself and agrees with h 0 for | z 2 | ≤ 1 . However, it is not necessarily volume preserving. But applying Moser's theorem (see [56]) to the image of the rectangles R 1 = [ -1 , 1] × [ -3 , -1] and R 2 = [ -1 , 1] × [1 , 3] which are the quadrilaterals Q 1 = ˜ h ( R 1 ) with endpoints ( -1 , -3) , (1 , -3) , (1 , 0) , ( -1 , -2) and Q 2 = ˜ h ( R 2 ) with endpoints (1 , 3) , ( -1 , 3) , ( -1 , 0) , (1 , 2) with the same size as R 1 and R 2 we infer that there is φ supported in Q 1 ∪ Q 2 such that h = φ ◦ ˜ h satisfies h ( Z (2) ) D = ˜ Z (2) D = U ([ -1 , 1] × [ -3 , 3]) .

E Identifiability and Approximation

In this section, we explain that approximation properties of function classes cannot be leveraged to obtain stronger identifiability results. The general setup is that we have two function classes G ⊂ F and we assume that G is dense in F (in the topology of uniform convergence on Ω ), i.e., for f ∈ F and any ϵ > 0 there is g ∈ G such that

We now investigate the relationship of identifiability results for mixings in either G or in F .

While this point is completely general, we will discuss this for concreteness in the context of nonlinear ICA and considering polynomial mixing functions for G . We also assume that F are all diffeomorphisms (onto their image). Suppose we have latents Z ∼ U ([ -1 , 1] d ) and observe a mixture X = h ( Z ) . We use the shorthand Ω = [ -1 , 1] d from now on. Let us suppose that h ∈ G . Then the following result holds.

Lemma 10. Suppose ˜ h ( Z ) D = X D = h ( Z ) where Z ∼ U (Ω) , h, ˜ h ∈ G and h satisfies an injectivity condition as defined in Assumption 2 in [2]. Then h and ˜ h agree up to permutation of the variables.

Proof. This is a consequence of Theorem 4 in [2]. Essentially, they show in Theorem 1 that h and ˜ h agree up to a linear map and then use independence of supports to conclude (in our simpler setting here, one could also resort to the identifiability of linear ICA).

The injectivity condition is a technical condition that ensures that the embedding h is sufficiently diverse, but this is not essential for the discussion here. Let us nevertheless mention here that it is not clear whether this condition is sufficiently loose to ensure identifiability in a dense subspace of F because it requires an output dimension depending on the degree of the polynomials.

We now investigate the implications for F . It is well known that ICA in general nonlinear function is not identifiable, i.e., for any f ∈ F there is a ˜ f ∈ F such that f ( Z ) D = ˜ f ( Z ) . To find such an ˜ f one use, e.g., the Darmois construction, radius dependent rotations or, more generally, measure preserving transformations m such that m ( Z ) D = Z . Then we have the following trivial lemma.

Lemma 11. For every ϵ > 0 there are functions g, ˜ g ∈ G such that

and then the Wasserstein distance satisfies W 2 ( f ( Z ) , g ( Z )) < ϵ , W 2 ( ˜ f ( Z ) , ˜ g ( Z )) < ϵ .

Remark 6. For the definition and a discussion of the Wasserstein metric we refer to [90]. Alternatively we could add a small amount of noise, i.e. X = f ( Z ) + ν and then consider the total variation distance.

Proof. The first part follows directly from the Stone-Weierstrass Theorem which shows that polynomials are dense in the continuous function on compact domains. The second part follows from the coupling Z → ( f ( Z ) , g ( Z )) and (113).

The main message of this Lemma is that whenever there are spurious solutions in the larger function class F we can equally well approximate the ground truth mixing f and the spurious solution ˜ f by functions in G . In this sense, the identifiability of ICA in G does not provide any guidance to resolve the ambiguity of ICA in F . So identifiability results in G are not sufficient when there is no reason to believe that the ground truth mixing is exactly in G .

Actually, one could view the approximation capacity of G also as a slight sign of warning as we will explain now. Suppose the ground truth mixing satisfies g ∈ G . Since ICA in F is not identifiable we can find ˜ f ∈ F which can be arbitrarily different from g but such that g ( Z ) = ˜ f ( Z ) . Then we can approximate ˜ f by ˜ g with arbitrarily small error. Thus, we find almost spurious solutions, i.e., ˜ g ∈ G such that W 2 ( ˜ g ( Z ) , g ( Z )) < ϵ for an arbitrary ϵ > 0 but ˜ g and g correspond to very different data representations, in this sense the identifiability result is not robust.

When only considering the smaller space G this problem can be resolved by considering norms or seminorms on G , for polynomials a natural choice is the degree of the polynomial. Indeed, suppose that we observe data generated using a mixing g ∈ G . Then there might be spurious solutions ˜ f ∈ F which can be approximated by ˜ g ∈ G but this approximation will generically have a larger norm (or degree for polynomials) than g . Therefore, the minimum description length principle [69] favors g over ˜ g .

This reasoning can only be extended to the larger class F if the ground truth mixing f (for the relevant applications) can be better approximated (i.e., with smaller norm) by functions in G than all spurious solutions ˜ f . This is hard to verify in practice and difficult to formalize theoretically. Nevertheless, this motivates to look for identifiability results for function classes that are known to be useful representation learners and used in practice, in particular neural nets. We emphasize that alternatively one can also directly consider norms on the larger space F penalizing, e.g., derivative norms, to perform model selection.

F Additional details on experimental methodology

In this appendix, we give more details about our experimental methodology. First, we derive the log-odds and prove Lemma 1 in Appendix F.1. Then, we use it to design our model and theoretically justify our contrastive approach in Appendix F.2, by connecting it to the Bayes optimal classifier. Then, we describe the NOTEARS regularizer in Appendix F.3, describe the limitations of the contrastive approach in Appendix G.1 and finally, in Appendix F.4, we describe the ingredients needed for an approach via Variational Autoencoders, the difficulties involved and how to potentially bypass them.

F.1 Proof of Lemma 1

Lemma 1. When we have perfect interventions, the log-odds of a sample x coming from X ( i ) over coming from X (0) is given by

for a constant c i independent of x .

Proof. We will write the log likelihood of X ( i ) = f ( Z ( i ) ) using standard change of variables. As we elaborate in Appendix A.1, we have

Also, denote by Θ ( i ) = ( B ( i ) ) ⊤ B ( i ) the precision matrix of Z ( i ) (see Appendix A.1 for full derivation). By change of variables,

where J f -1 denotes the Jacobian matrix of partial derivatives. Let s (1) , s (2) , . . . , s ( d ) denote the rows of B (0) . We then compute the log-odds with respect to X (0) , the base distribution without interventions (and where µ (0) = 0 ) to get

for a constant c i independent of z . For the second equality, we used Lemma 3.

F.2 A theoretical justification for the log-odds model

In this section, we provide more details on our precise modeling choices for the contrastive approach, and show theoretically why it encourages the model to learn the true underlying parameters.

Recall that, motivated by Lemma 1, we model the log-odds as

where h ( · , θ ) denotes a neural net parametrized by θ , parameters w ( i ) are the rows of a matrix W and α i , β i , γ i are learnable parameters. Moreover, we have g 0 ( x ) = 0 as we compute the log-odds with respect to X (0) . Therefore, the cross entropy loss given by

Let C ( x ) ∈ { 0 , 1 } denote the label of the datapoint x indicating whether it was an observational datapoint or an interventional datapoint respectively. By using the cross entropy loss, we are treating the model outputs as logits, therefore we can write down the posterior probability distribution using a logistic regression model as

Moreover, by using a sufficiently wide or deep neural network with universal approximation capacity and sufficiently many samples, our model will learn the Bayes optimal classifier, as given by Lemma 1. Therefore, we can equate the probabilities to arrive at

implying

This suggests that in optimal settings, our model will learn (up to scaling)

thereby inverting the nonlinearity and learning the underlying parameters.

F.3 NOTEARS [102]

In our algorithm, using the parameters w ( i ) as rows, we learn the matrix W , which we saw in optimal settings will be B = D -1 / 2 (Id -A ) . Note that the off-diagonal entries of B form a DAG. Therefore, the graph encoded by W 0 , which is defined to be W with the main diagonal zeroed out, must also be a DAG.

However, in our algorithm, we don't explicitly enforce this acyclicity. There are two ways to get around this. One way is to assume a default causal ordering on the Z i s, which is feasible as we don't directly observe Z . Then, we can simply enforce W 0 to be triangular and train via standard gradient descent and W 0 is guaranteed to be a DAG. However, this approach doesn't work because the interventional datasets are given in arbitrary order and so we are not able to match the datasets to the vertices in the correct order. So this merely defers the issue.

Instead, the other approach that we take in this work is to regularize the learnt W 0 to model a directed acyclic graph. Learning an underlying causal graph from data is a decades old problem that has been widely studied in the causal inference literature, see e.g. [12, 102, 64, 80, 4] and references therein. In particular, the work [102] proposed an analytic expression to measure the DAGness of the causal graph, thereby making the problem continuous and more efficient to optimize over.

A matrix W ∈ R d × d is a DAG if and only if

where ◦ is the matrix Hadamard product and exp is the matrix exponential. Moreover, R has gradient

Therefore, we add R NOTEARS ( W ) = tr exp( W 0 ◦ W 0 ) -d as a regularization to our loss function. As in prior works, we could also consider the augmented Lagrangian formulation however it leads to additional training complexity. There have been several follow-up works to NOTEARS such as [98] (see [4] for an overview) that suggest different regularization schemes. We leave exploring such alternatives to future work.

F.4 A Variational Autoencoder approach

In this section, we describe the technical details involved with adapting Variational Autoencoders (VAEs) [39, 68] for our setting. We highlight some difficulties that we will encounter and also suggest possible ways to work around them.

Indeed, most earlier approaches for causal representation learning have relied on maximum likelihood estimation via variational inference. In particular, they have relied on autoencoders or variational autoencoders [36, 2]. In our setting, V AEs are a viable approach. More concretely, we can encode the distribution X (0) to ϵ , then apply the B ( i ) parameter matrix before finally decoding to X ( i ) . To handle the fact that we don't have paired interventional data as in [6], we could potentially modify the Evidence Lower Bound (ELBO) to include some divergence measure between the predicted and measured interventional data. Finally, this can be trained end-to-end as in traditional V AEs. We now describe this in more detail.

We will use VAEs to encode the observational distribution X (0) to ϵ , so the encoder will formally model B (0) ◦ ( f -1 ( . ) -µ ( i ) ) where we use the notation from Appendix A.1. Note here that we cannot directly design the encoder to map X to Z because we do not know the prior distribution on Z . Indeed, the objective is to learn it.

Let I denote the set of intervention targets. Assume the targets are chosen uniformly at random, which can be done in practice by subsampling each interventional distribution to have the same size. Suppose we had paired counterfactual data D = ∪ i ∈ I D i of paired interventional datasets, i.e. D i consists of pairs ( X (0) , X ( i ) ) with corresponding probability density denoted p ( i ) ( x (0) , x ( i ) ) . Following [39, 101], define an amortized inference distribution as q ( ϵ | x (0) ) . Then, we can bound the expected log-likelihood as

(128)

where the last term is the standard Evidence Lower Bound (ELBO). This can then be trained end-toend via the reparameterization trick. A similar approach was taken in [6] who had access to paired counterfactual data.

However, we do not have access to such paired interventional data, but instead we only observe the marginal distributions X ( i ) . To this end, conditioned on ϵ , we can split the term ln p ( i ) ( x (0) , x ( i ) | ϵ ) = ln ˜ p (0) ( x (0) | ϵ ) + ln ˜ p ( i ) ( x ( i ) | ϵ ) where ˜ p ( i ) denotes the density of the intervened marginal of p ( i ) , which corresponds to using the interventional decoder f ◦ (( B ( i ) ) -1 ( . ) + µ ( i ) ) . Nevertheless, E ϵ ∼ q ( x (0) ) [ln ˜ p ( i ) ( x ( i ) | ϵ )] is still not tractable in unpaired settings. As a heuristic, we could consider a modified ELBO by modifying this term to measure some sort of divergence metric between the true interventional data X ( i ) and the generated interventional data ˜ p ( i ) ( x ( i ) | ϵ ) , similar to [101]. We leave it for future work to explore this direction.

G Additional experimental results

In this section, we provide additional experimental results and discussions to explore the effect of shift strength of the interventions, data scale, and noise distribution.

G.1 Limitations of the contrastive approach

0.0

0.5

1.0

10

20

SHD

0.0

0.5

1.0

0.8

0.9

1.0

AUROC

0.0

0.5

1.0

0.5

1.0

MCC

0.0

0.5

1.0

0.0

0.5

1.0

R

2

As mentioned in the main part of the paper, our contrastive algorithm struggles to recover the ground truth latent variables when considering interventions without shifts. We illustrate the dependence on the shift strength in Figure 5. In this section, we provide some evidence why this is the case. Apart

2

0

2

z

0.6

0.8

1.0

CE

2

0

2

z

2

4

from setting the stage for future works, this also enables practitioners to be aware of potential pitfalls when applying our techniques.

5

0

5

Z

1

0.0

0.1

0.2

0.3

0.4

Ground truth

Z

1

Recovered

Z

1

5.0

2.5

0.0

2.5

Z

1

0.0

0.1

0.2

0.3

0.4

Ground truth

Z

1

Recovered

Z

1

2

0

2

Z

1

2

1

0

1

2

3

4

Z

1

2

0

2

Z

1

2

1

0

1

2

3

Z

1

We suspect that the main reason for this behavior is the non-convexity of the parametric output layer. Note that this is very different from the well known non-convexity of (overparametrized) neural networks. While neural networks parametrize non-convex functions their final layer is typically a convex optimization problem, e.g., a least squares regression or a logistic regression on the features.

Moreover, it is well understood that convergence to a global minimizer using gradient descent is possible under suitable conditions, see, e.g., [47].

This is different for our quadratic log-odds expression where gradient descent is not sufficient to find the global minimizer. To illustrate this we consider the case of a single latent, i.e., d = 1 , then the ground truth parametric form of the log-odds in 6 can be expressed as

for some constants a , b , c and b = 0 if η (1) = 0 , i.e., there is no shift. We moreover assume, for the sake of argument, that the final layer implementing h ( z ) = a ( z -b ) 2 + c is fixed to the ground truth parametric form of the log-odds (then there is also no scaling ambiguity left). Now, let us fix a point x 0 and define z 0 = f -1 ( x 0 ) and

Then the (sample conditional) cross entropy loss as a function of ˆ z = ˆ f -1 ( x 0 ) for our learned function ˆ f is given by

̸

We here drop the intervention index, since we focus on a one dimensional illustration. We plot two examples of such loss functions for b = 0 and b = 0 in Figure 6.

To verify that this is indeed the reason for our failure to recover the ground truth latent variables, we investigate the relation between estimated latent variables and ground truth latent variables. The result can be found in Figure 7 and, as we see, when η = 0 , the distribution of the recovered latents is skewed. Moreover, we find that we essentially recover the latent variable up to a sign, i.e., ˆ Z = | Z | (there is a small offset because we enforce ˆ Z to be centered). Note that this explains the tiny MCC scores in Table 1, since E ( Z | Z | ) = 0 for symmetric distributions. This observation might also be a potential starting point to improve our algorithm. For non-zero shifts we recover the latent variables almost perfectly.

G.2 Varsortability and data scale

An important observation in causal discovery was that many benchmark settings exhibit specific correlation structures that can be exploited to infer causal directions. The most prominent example is varsortability [66] which measures how well the causal order agrees with the order induced by the magnitude of the variance. This is a measure between 0 and 1 and for values close to 0 and 1 the variances contain a lot of information about the causal order while close to 0 . 5 this is not the case. It was shown in [66] that typical parameter choices exhibit high varsortability ≈ 1 which can be exploited by trivial algorithms. A related notion is R 2 -varsortability [67] which is a similar notion where the variance of the variable is replaced by the residual variance after standardizing the variables and then regressing out all other variables, i.e., the unexplained variance.

In our setting the varsortability of the latent variables Z is quite high (see [66] for the full definition), e.g., for d = 10 and k = 2 average varsortability is 0 . 92 ± 0 . 02 . Note that it is not obvious how this can be exploited algorithmically. Indeed, we only observe a mixture X = f ( Z ) and the scale of Y is not identifiable. Nevertheless, to rule out that we implicitly use the scale of the variables we ran the same experiments except that we standardized the latent variables Z of the observable distribution (and apply the same scaling to the interventional distributions). The results can be found in Table 4 and are roughly similar to the result in Table 2 for the same settings without standardization. Note that R 2 -sortability is not substantially different from . 5 (roughly 0 . 4 ) in our settings, thus the R 2 -sortability cannot be exploited to infer the causal structure, even when observing Z .

G.3 Effect of noise distribution

Our theoretical results only cover the setting of Gaussian SCMs and we even show that identifiability does not necessarily hold for uniformly distributed noise (see Appendix D.2). Nevertheless, we can

evaluate the performance of our algorithm for linear SCMs with non-Gaussian noise distribution. We rerun one of the settings in Table 2 for different noise distributions. The results can be found in Table 5, and they show a slightly worse but still reasonable performance than for Gaussian data. This is in line with the observation that using a Gaussian likelihood for non-Gaussian data gives good results in, e.g., causal discovery.

H Hyperparameters, architectures and further additional details on experiments

In this section, we give additional details on the experiments. We use PyTorch [60] for all our experiments.

Data generation To generate the latent DAG and sample the latent variables we use the sempler package [21]. We always sample ER( d, k ) graphs and the non-zero weights are sampled from the distribution w ij ∼ U ( ± [0 . 25 , 1 . 0]) . To generate linear synthetic data, we sample all entries of the mixing matrix i.i.d. from a standard Gaussian distribution. For non-linear synthetic data f is given by the architecture in Table 9 with weights sampled from U ([ -1 √ in feat , 1 √ in feat ]) . For image data, similar to [2], the latents Z are the coordinates of balls in an image (but we sample them differently as per our setting) and a 64 × 64 × 3 RGB image is rendered using Pygame [77] which encompasses the nonlinearity. Other parameter choices such as dimensions, DAGs, variance parameters, and shifts are already outlined in Section 6 but for clarity we also outline them in Tables 6, 7, and 8.

Models For synthetic data, the model architecture for h ( x, θ ) is a one hidden-layer MLP with 512 hidden units and LeakyReLU activation functions. For the parametric final layer, we decompose the matrix W as W = D (Id -A ) where D is a diagonal matrix and A is a matrix with a masked diagonal and they are both learned. This factorization shall improve stability. We initialize A = 0 and D = Id . Also, the sparsity penalty and the DAGness penalty are only applied to A . For the baseline VAE we use the same architecture for both encoder and decoder. The code for the linear baseline is from [84].

For image data, the model architectures for h ( x, θ ) are small convolutional networks. For the contrastive approach the architecture can be found in Table 10. For the VAE model, we use the encoder architecture as in Table 11 and the decoder architecture as in Table 12 respectively.

Training For training we use the hyperparameters outlined in Table 13. We use a 80 -20 split for train and validation/test set respectively. After subsampling each dataset to the same size, we copy the observation samples for each interventional dataset in order to have an equal number of observational and interventional samples so they can be naturally paired during the contrastive learning. We select the model with the smallest validation loss on the validation set, where we only use the cross entropy loss for validation. For the V AE baseline we use the standard V AE validation loss for model selection.

Compute and runtime The experiments using synthetic data were run on 2 CPUs with 16Gb RAM on a compute cluster. For image data we added a GPU to increase the speed. The runtime per epoch for the synthetic data and 10000 samples was roughly 10s. The total run-time of the reported experiments was about 100h (i.e. 200 CPU hours) and 20h on a GPU, but hyperparameter selection and preliminary experiments required about 10 times more CPU compute.

Mean Correlation Coefficient (MCC) Mean Correlation Coefficient (MCC) is a metric that has been utilized in prior works [37, 41] to quantify identifiability. MCC measures linear correlations up to permutation of the components. To compute the best permutation, a linear sum assignment problem is solved and finally, the correlation coefficients are computed and averaged over. A high MCCindicates that the true latents have been recovered. In this work, we compute the transformation using half of the samples and then report the MCC using the other half, i.e. the out-of-distribution samples.

Further experimental attempts and dead Ends We will briefly summarize a few additional settings we tried in our experiments.

We fixed D = Id in the parametrization of W . This fixes the scaling of the latent variables in some non-trivial way. This resulted in very similar results.

We tried to combine the contrastive algorithm with a VAE approach. Since the ground truth latent variables Z are highly correlated, they are not suitable for an autoencoder that typically assumes a factorizing prior. Therefore, we map the estimated latent variables for the observational distribution to the noise distribution ϵ which factorizes. Note that for the ground truth latent variables the relation ϵ = B (0) Z (0) holds so we consider ˆ ϵ = W ˆ Z . We

use ˆ ϵ as the latent space of an autoencoder on which we then stack a decoder. We train using the observational samples and the ELBO for the VAE part and with the contrastive loss (and interventional and observational samples) for the ˆ Z embedding. This resulted in a slightly worse recovery of the latent variables (as measured by the MCC score) but much worse recovery of the graph. We suspect that this originates from the fact that the matrix W is used both in the parametrization of the log-odds and to map ˆ Z to ϵ which might make the learning more error-prone. Note that this is different from the suggestion in Section F.4.

Choosing a larger learning rate for the parametric part than for the non-parametric part seemed to help initially, but using the same sufficiently small learning rate for the entire model was more robust.

We tried larger architectures for both synthetic and image data. They turned out to be more difficult to train without offering any benefit.

For image data we often observed overfitting when using smaller sample sizes. The overfitting persisted even when we used a pre-trained ResNet [24] with frozen layers. We suspect that for image data the performance (in particular the sample complexity) can be further improved by carefully tuning regularization and model size.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of learning latent causal models from interventional data. The problem was formulated under linear or polynomial mixing in previous works and this work considers a more general setting of nonlinear mixing. The main contributions are the identifiability results of the latent causal model and a contrastive algorithm that identifies the latent model.\n\nSTRENGTHS:\n1. The extension to nonlinear mixing is an important and challenging problem.\n\n2. Using the contrastive algorithm to identify the model parameters is novel and sound.\n\nWEAKNESSES:\n1. Single-node interventions are considered, while more nodes can be intervened in each environment in practice. For theoretical results, I think focusing on single-node intervention is fine, but it should be discussed whether the results have the potential to be generalized. \n\n2. It was not mentioned whether the latent dimension $d$ is identifiable in the nonlinear setting. For all the identifiability results in the paper, the considered $\\widetilde{Z}^{(i)}$ is assumed to have the same dimension as the true latent variable $Z^{(i)}$ (i.e., $d$). Note that $d$ is identifiable and it equals to the rank of the precision matrix of X in the linear setting. (Section 3.1 [1]). \n\nThis issue occurs in the experiments as well.  $d$ is provided to the contrastive method. But it is often not possible to know $d$ in practical settings.\n\n[1] C. Squires, A. Seigal, S. Bhate, and C. Uhler. Linear causal disentanglement via interventions, 2023.\n\nAdditional comment: I think it would be helpful to provide a concrete toy example to demonstrate the identifiability since the proof intuition is a bit vague.\n\nQUESTIONS:\nWhether $d$ is identifiable in the nonlinear setting? And how to identify $d$ for the contrastive method?\n\nI would like to raise my score if this problem can be addressed properly.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to identify latent variables via a nonlinear mixing function interventional data. The authors prove strong identifiability results for unknown single-node interventions, extending previous work that focused on linear maps, polynomial mixing functions or paired counterfactual data. The paper proposes an interesting contrastive algorithm to identify the latent variables and evaluates its performance on some synthetic tasks and a simple image dataset modified from Ahuja et al [2023]. While there have been other very recent works that examine the interventional setting, this is the first I'm aware of that only requires $d$ interventions to recover $d$ latents with minimal constrains on the mixing function.\n\nSTRENGTHS:\n* This is the strongest theoretical result that I am aware of in causal representation learning. The gaussian assumption is obviously restrictive, but other than that, the paper proves identifiability for a very practical class of problems: they only need $d$ interventions to identify $d$ latents, and make no further restrictive assumptions on the mixing function beyond standard infectivity / diffeomorphism assumptions. \n* The paper is very clearly written both in the main text and the appendix. \n* The contrastive algorithm that they propose to implement their approach is interesting. It has some optimization issues which the paper is upfront about, but I'm still curious to see how it would work on larger problems.\n\nWEAKNESSES:\nI don't have many complaints, but I do have some nitpicks.\n* In the introduction (particularly the first paragraph), the paper makes it sound like the generative process for the data is via a neural network (transformers & diffusion models in line 16, and a similar comment is made in line 124 in support of Assumption 1). While we can build generative models of complex data with neural networks, that is not the $f(\\cdot)$ that we care about in causal representation learning. The real generative function, $f(\\cdot)$, is a property of the world---it's the camera or microscope that photographs a scene---and we don't have control over that. I realize this makes the diffeomorphism assumptions unrealistic, but I think it's better to just be upfront about that as a limitation. \n* I have a similar complaint about the defence of the Gaussian assumption in lines 138 - 141: real processes almost certainly are not Gaussian, so we should treat it as a model of the world and be upfront about that (in the Box, \"all models are wrong\" sense). The paper would be strong if you were upfront about the fact that it is clearly restrictive but useful (because it gives strong identifiability results), and then evaluated sensitivity of the method to non-gaussian latents in the experiments. \n\nMore minor:\n* Line 32 - I believe causal representation learning, but it remains to be seen whether it is either necessary or sufficient to build trustworthy systems, so that's a strong claim to make.\n\nQUESTIONS:\n* Have you experimented with non-gaussian latents? What happens?\n* How well does the method work on larger problems (e.g. 5 - 10 balls? Or more?)\n\nLIMITATIONS:\nAs mentioned above, the paper could be a little more upfront about the implication of the assumptions it makes, but overall it does a good job of addressing limitations. I also liked the \"deadends\" section at the end of the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the task of causal representation learning (causal disentanglement) from interventions, with (1) a linear latent structural causal model, (2) a nonlinear mixing function, and (3) single-node interventions. They show that, under perfect interventions, the generative model is identifiable up to trivial indeterminacies. They propose a method based on contrastive learning for recovering the generative model, and show that their method works well in practice (outperforming a baseline which only allow for linear mixing).\n\nSTRENGTHS:\n### Significance\nThis paper presents a significant theoretical contribution to an established line of work, while introducing techniques and connections that will be useful for future works. In particular, the advance from linear (or polynomial) mixing to general non-linear mixing is a big step towards realistic generative modeling. On the practical side, they also develop a solid contribution by showing that contrastive learning can be used to disentangle the latent variables. \n\n### Clarity\nThe paper is very clear in describing its contribution and how it compares to previous work. The presentation is quite thorough: they clearly describe their setup and discuss their assumptions, they show that their sufficient conditions for identifiability are actually necessary conditions, they discuss the unsuitability of polynomial mixing, and they provide a detailed description of their experimental methodology with accompanying code.\n\nWEAKNESSES:\nThere are no major weaknesses. There are some relatively minor points of confusion / potential typos and some small suggestions to improve the paper in the **Questions** section.\n\nIf I had to pick a weakness, it would be that the proposed algorithm suffers from problems with local optima. It would be ideal to have an algorithm which provably recovers the latent representation, and some sample complexity results. These results would make the paper feel very \"complete\", and with these results the paper would probably be a better fit for a journal.\n\nQUESTIONS:\n### Questions\n1. In Equation (3), why are we able to identify shifts without scaling indeterminacy? This seems odd: if we scale a variable, I would expect that the shift also scales.\n2. In Equation (7), should $h$ in the last term have a subscript? Then it is not a matrix so I don't see how it should be in the inner product.\n\n### Suggestions\n1. In line 26, the cited papers are about independent component analysis, and these are given as examples of identifiability in causal representation learning (CRL). I agree that ICA can be cast as a special case of CRL. However, I have the feeling that for the purposes of clarity, it might be best to use a different umbrella term like \"identifiable representation learning\"; there is nothing really \"causal\" about ICA.\n2. Adapt the synthetic data generation process to reduce varsortability and $R^2$-sortability [1,2]. One procedure which takes these issues into account is given by [3]. \n\n[1] Reisach, A., Seiler, C., & Weichwald, S. (2021). Beware of the simulated DAG! Causal discovery benchmarks may be easy to game.\n\n[2] Reisach, A. G., Tami, M., Seiler, C., Chambaz, A., & Weichwald, S. (2023). Simple sorting criteria help find the causal order in additive noise models.\n\n[3] Squires, C., Yun, A., Nichani, E., Agrawal, R., & Uhler, C. (2022). Causal structure discovery between clusters of nodes induced by latent factors.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn recent years, the theory of non-linear independent component analysis and causal representation learning has witnessed a lot of interesting developments. In this work, the authors study the problem of causal representation learning in the presence of interventional datasets, where interventions occur on the latents. The authors show that when the latents follow a linear structural causal model with a Gaussian distribution, then under general non-linear mixing (diffeomorphisms) the latents can be recovered up to permutation and scaling. For imperfect interventions, the authors show that it is still possible to recover the partial order under topological ordering of the underlying causal graph G can be recovered.  The authors present a new method based on contrastive learning that learns to distinguish interventional data from observational data and test it out on some synthetic datasets.\n\nSTRENGTHS:\nThe authors have studied an important problem in the area of intereventional causal representation learning. The results proposed in the work for the case of perfect interventions show that strong identification is achievable with perfect interventional data. The authors also make progress on the difficult problem of tackling imperfect interventions. The theory of the paper is overall quite insightful.\n\nWEAKNESSES:\nI have a question and a claim about weakness. Let us consider the following setting. \n\nData generation for obervational data is given as:\n\n$\\epsilon \\sim \\mathcal{N}(0, \\sigma^2 Id)$, \n$z = A \\epsilon$, \n$x  = f(z)$\n\nwhere $A$ is invertible, $\\mathcal{N}$ is normal distribution, $\\epsilon$ is noise (each component is independent), $f(\\cdot)$ is injective. \n\nData generation in interventional environment $k$ where $p^{th}$ component is intervened is given as \n\n$\\epsilon^{(k)} \\sim \\mathcal{N}(\\mu_p e_p, \\sigma^2 Id + \\sigma_p^2 e_p e_p^{\\top})$, $z^{(k)}= A \\epsilon^{(k)}$, $x = f(z^{(k)})$\n\nwhere $e_p$ is a vector with zeros everywhere except at $p^{th}$ entry, $Id$ is identity, $\\mu_p$ is mean under intervened distribution\n\nWe index the observational data as $0$ and interventional data from $1$ to $p$.\n\nIf we condition on the index of the data, the different components of $\\epsilon$ are conditionally independent and follow an exponential distribution.  We can now leverage the theory in i-VAE applied to identifying $\\epsilon$ and mixing function $f \\circ A$.  In this case, if we have $p\\geq 2d$ and sufficient variability condition from Theorem 1 in http://proceedings.mlr.press/v108/khemakhem20a/khemakhem20a.pdf is satisfied, then we achieve permutation and scaling identification.  \n\n\nTherefore, we can assume to have identified $\\epsilon$ up to permutation and scaling. We call this estimate $\\hat{\\epsilon} = P \\Lambda z$, where $P$ is permutation, $\\Lambda$ is diagonal. Note that $\\hat{\\epsilon} =\\Lambda^{-1}P^{-1} A^{-1}z$. We now have a linear relationship between observed $\\hat{\\epsilon} $ and underlying true $z$. \n\nWe can now leverage imperfect intervention results under linear mixing from https://arxiv.org/pdf/2301.08230.pdf (Theorem 13, see Table 1) to achieve mixing consistency based identification of $z$. \n\nWe can even reduce the number of interventional distributions needed from $2d$ to $d$ provided we assume variance in Normal distribution is known by leveraging results from https://proceedings.mlr.press/v177/lachapelle22a/lachapelle22a.pdf. \n\nThe characterization I describe above suggests that it is possible to get identification results by combining i-VAE and  https://arxiv.org/pdf/2301.08230.pdf in a straightforward way. Either I am missing something? I would like the authors to clarify if they considered this simple combination? Further, if the authors agree with above characterization, how do the authors modify their contributions in this light?\n\n\nI don't quite agree with line 253, distributional assumptions. I think Gaussians is still a strong assumption and authors would benefit by being upfront about it.\n\nQUESTIONS:\nPlease see the weakness section above. My final score depends on clarifcation to the questions I raised above.\n\nLIMITATIONS:\nThe authors would benefit by having a discussion on the limitations. For instance, they should talk about why is Gaussian assumption limiting as an example.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}