Identifying Weight-Variant Latent Causal Models
Authors: Yuhang Liu
Zhen Zhang
Dong Gong
Mingming Gong
Biwei Huang
Anton van den Hengel
Kun Zhang
Javen Qinfeng Shi
Venue: Submitted to ICLR 2023 (2023)

Abstract: The task of causal representation learning aims to uncover latent higher-level causal representations that affect lower-level observations. Identifying true latent causal representations from observed data, while allowing instantaneous causal relations among latent variables,  remains a challenge, however. To this end, we start from the analysis of three intrinsic properties in identifying latent space from observations: transitivity, permutation indeterminacy, and scaling indeterminacy. We find that transitivity acts as a key role in impeding the identifiability of latent causal representations. To address the unidentifiable issue due to transitivity, we introduce a novel identifiability condition where the underlying latent causal model satisfies a linear-Gaussian model, in which the causal coefficients and the distribution of Gaussian noise are modulated by an additional observed variable. Under some mild assumptions, we can show that the latent causal representations can be identified up to trivial permutation and scaling. Furthermore, based on this theoretical result, we propose a novel method, termed Structural caUsAl Variational autoEncoder, which directly learns latent causal representations and causal relationships among them, together with the mapping from the latent causal variables to the observed ones. We show that the proposed method learns the true parameters asymptotically. Experimental results on synthetic and real data demonstrate the identifiability and consistency results and the efficacy of the proposed method in learning latent causal representations.

Under review as a conference paper at ICLR 2023
IDENTIFYING WEIGHT- VARIANT LATENT CAUSAL MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
The task of causal representation learning aims to uncover latent higher-level causal representations that affect lower-level observations. Identifying true latent causal representations from observed data, while allowing instantaneous causal relations among latent variables, remains a challenge, however. To this end, we start from the analysis of three intrinsic indeterminacies identifying latent space from observations: transitivity, permutation indeterminacy, and scaling indeterminacy. We find that transitivity acts as a key role impeding the identifiability latent causal representations. To address the unidentifiable issue due to transitivity, we introduce novel identifiability condition where the underlying latent causal model satisfies linear-Gaussin model, in which the causal coefficients and the distribution of Gaussian noise are modulated by an additional observed variable. Under some mild assumptions, we can show that the latent causal representations can be identified up to trivial permutation and scaling. Furthermore, based on this theoretical result, we propose novel method, termed Structural caUsAl Variational autoEncoder (SuaVE), which directly learns latent causal representations and causal relationships among them, together with the mapping from the latent causal variables to the observed ones. We show that SuaVE learns the true parameters asymptotically. Experimental results on synthetic and real data demonstrate the identifiability and consistency results and the efficacy of SuaVE in learning latent causal representations.
INTRODUCTION
While there is no universal formal definition, one widely accepted feature of disentangled repre- sentations (Bengio et al., 2013) is that a change in one dimension corresponds change in one factor of variation in the underlying model of the data, while having little effect on others. The underlying model is rarely available for interrogation, however, which makes learning disentangled representations challenging. Several excellent works for disentangled representation learning have been proposed that focus on enforcing independence over the latent variables that control the factors of variation (Higgins al., 2017| Chen al. 2018 Locatello et al. 2019 Kim & Mnih 2018 Locatello et al.] 2020). In many applications, however, the latent variables are not statistically independent, which is at odds with the notion of disentanglement above, i.e., foot length and body height exhibit strong positive correlation in the observed data (Träuble et al. (2021).
Causal representation learning avoids the aforementioned limitation, tion that exposes the unknown high-level causal structural variables, and the relationships between them, from set of low-level observations (Schölkopf et al. [02II. Unlike disentangled representa- tion learning, it identifies the possible causal relations among latent variables. In fact, disentangled representation learning can be viewed as a special case of causal representation learning where the latent variables have causal influences (Schölkopf et al., (021). One of the most prominent additional capabilities of causal representations is the ability to represent interventions and to make predictions regarding such interventions (Pearl 2000), which enables the generation of new samples that do not lie within the distribution of the observed data. This can be particularly useful to improve the generalization of the resulting model. Causal representations also enable answering counterfactual questions, e.g., would a given patient have suffered heart failure if they had started exercising a year earlier (Schölkopf et al. 2021)
aims to learn a representa-

---

Under review as a conference paper at ICLR 2023
Despite its advantages, causal representation learning is notoriously hard problem-witte certain assumptions, identifying the true latent causal model from observed data is generally not possible. There are three primary approaches to achieve identifiability: 1) adapting (weakly) supervised
methods with given latent causal graphs or/and labels (Kocaoglu et al. 2018: Yang et al., 2021
Von Kügelgen al. 202T: Brehmer al. 2022 2) imposing sparse graphical conditions, e.g. with
bottleneck graphical conditions (Adams et al. 2021;
1 Xie et al. 2020 Lachapelle et al. 2021). 3)
using temporal information (Yao et al. 2021 Lippe et al. (022).
brief review is provided in Section
2 For the supervised approach, when labels are known, the challenging identifiability problem in
latent space has been transferred to an identifiability problem in the observed space, for which some commonly-used functional classes have been proven to be identifiable (Zhang & Hyvarinen 2012
Peters et al. 2014). Given latent causal graphs overly depends on domain knowledge. For the second approach, many true latent causal graphs do not satisfy the assumed sparse graph structure. The
temporal approach is only applicable when temporal information or temporal intervened information among latent factors is available.
In this work, we explore a new direction in the identifiability of latent causal representations, by allowing causal influences among latent causal variables to change, motivated by recent advances in nonlinear ICA (Hyvarinen et al.] 2019] Khemakhem et al.] 2020). (Hyvarinen et al,, [201)] Khemakhem al., 2020) have shown that with additional observed variable to modulate latent independent variables, the latent independent variables are identifiable. Then a question naturally arises: with causal relationships among latent variables, what additional assumptions are required for the identifiability? To answer this question, we start from the analysis of three intrinsic indeterminacies in latent space (see Section 3) transitivity, permutation indeterminacy, and scaling indeterminacy. which further give rise to the following insights. 1) Transitivity is the scourge of identifiablity of the latent causal model. 2) Permutation indeterminacy means the recovered latent variables can have arbitrary permutation of the underlying orders, due to the flexibility in latent space. This nature enables us to enforce the learned causal representations in correct causal orders regularized by a predefined directed acyclic supergraph, avoiding troublesome directed acyclic graph (DAG) constraint. 3) Scaling and permutation indeterninacy only allow recovering latent causal variables up to permutation and scaling, not the exact values. To overcome the transitivity challenge we model the underlying causal representation with weight-variate linear Gaussian models, where both the weights (i.e., causal coefficients) and the mean and variance of the Gaussian noise are modulated by an additional observed variable u (see Section 4. With these assumptions, we can show that the latent causal representations can be recovered up to a trivial permutation and scaling. The key to the identifiability is that the causal influences (weights) among the latent causal variables are allowed change. Intuitively, the changing causal influences enable us obtain interventional observed data, which enables the identifiability of the latent causal variables. Based on this result, in Section 5 we further propose a novel method, Structural caUsAl Variational autoEncoder (Sua VE), for learning latent causal representations with consistency guarantee. Section 6 verifies the efficacy of the proposed approach on both synthetic and real fMRI data.
RELATED WORK
Due to the challenges of identifiability in causal representation learning, most existing works handle this problem by imposing assumptions. We thus give a brief review of the related work on this basis.
(Weakly) Supervised Causal Representation Learning Approaches falling within this class assume known latent causal graphs or labels. CausalGAN (Kocaoglu et al. 2018) requires a-priori knowledge of the structure of the causal graph of latent variables, which is significant practical limitation.
CausalVAR (Yang et al., (021) needs additional labels to supervise the learning of latent variables.
Such labels are not commonly available, however, and manual labeling can be costly and error-prone. Von Kügelgen et al. (2021) use a known but non-trivial causal graph between content and style factors to study self-supervised causal representation learning. Brehmer et al.) (2022) Iearn causal representation in weakly supervised setting whereby they assume access to data pairs representing the system before and after a randomly chosen unknown intervention.
Sparse Graphical Structure Most recent progress in identifiability focuses sparse graphical structure constraints (Silva et al. 20066) Shiiizu et al. 2009 Anandkumar et al. 2213) Frot et al. 2019 Cai et al. 2019 Xie et al.
2020 2022h. Adams et al. (022) provided unifying viewpoint of
2

---

Under review as a conference paper at ICLR 2023
this approach whereby sparser model that fits the observation is preferred. However, they primarily consider linear relations between latent causal variables, and between latent causal variables and observed variables, which are often violated in real-world applications. Lachapelle et al. (202T) handles nonlinear causal relation between latent variables by assuming special sparse graphical structures. However, many latent causal graphs in reality may be more or less arbitrary, beyond purely sparse graph structure. In contrast, our work assumes a function class of latent variables, and does not restrict the graph structure over them.
Temporal Information The temporal constraint that the effect cannot precede the cause has been used repeatedly in latent causal representation learning (Yao et al. 2021 Lippe et al. [022] Yao et al.)
2022). For example, Yao et al. (2021) recover latent causal vanialess and the relations between them using Variational AutoEncoders and enforcing constraints in causal process prior. Lippe et al. (2022) learn causal representations from temporal sequences, which requires the underlying causal factors to be intervened. All of these works can be regarded as special cases of exploring the change of causal influences among latent variables in time series data. The approach proposed is more general since the observed auxiliary variable u could represent time indices, domain indices, or almost any additional side information.
Besides, Kivva et al. (2021) considers a nonlinear setting for latent causal discovery. assumes discrete latent causal variables that are rendered identifiable by mixture oracle, while in this work we consider continues latent causal variables.
3 THREE INDETERMINACIES IN LATENT CAUSAL MODELS
In this section, we first build a connection between nonlinear ICA and causal representation learning, by exploiting the correspondence between the independence of latent variables in nonlinear ICA and the independence of latent noise variables in causal representations. We then consider three indeterminacies in latent space: transitivity, permutation indeterminacy, and scaling indeterminacy, and analyse their impact on identifiability.
3.1 RELATING CAUSAL REPRESENTATION LEARNING WITH IDENTIFIABLE NONLINEAR ICA
Causal representation learning aims uncover latent higher-level causal representations that can explain lower-level raw observations with nonlinear mapping. Specifically, we assume that the observed variables x influenced by latent causal variables 2,, and the causal structure among 2; can be any directed acyclic graph (which is unknown). For each latent causal vari- able 2;, there is corresponding latent noise variable ns, as shown in Figure [ which represents some un- measured factors that influence 2,. The latent noise variables n, are assumed to be independent] [ with each other, conditional on the observed variable u, in causal system (Peters et al.] 2017), so it is natural to leverage recent progress in nonlinear ICA (Hyvarinen al.) 2019: Khemakhem et al. 2020), which has shown that the independent latent noise variables ni are identifiable under relatively mild assumptions, if they are modulated by an additionally observed variable For example, if (ni) are Gaussian distributed with their mean and variance modulated by an observed variable u, and the mapping from n to z is invertible. Taking one step further, our goal is recover the latent causal variables 2,. However, with the assumptions for the identifiability of latent noise variables the latent causal variables 2,. The reason of such con-identifiability will be given in Section 3.22 Given this fact, a further question is what additional conditions are needed to recover the latent causal variables z. The corresponding identifiability conditions will be given in Section 4
Camel
Figure 1: Causal representation with u.
from nonlinear ICA, it is still insufficient to identify
For convenience in the later parts, with slight abuse of definition, independent n; means that are mutually independent conditional the observed variable
3

---

Under review as a conference paper at ICLR 2023
3.2 TRANSITIVITY: THE CHALLENGE OF IDENTIFYING LATENT CAUSAL REPRESENTATIONS
Even with the identifiable nu, it is impossible to identify the latent causal variables 2 without additional assumptions. To interpret this point, for simplicity, let us only consider the influences the graph structure in Figure [ assume that 21 = Ri, 22 = 21 + R2 and x ;= f(21, 22) + E (case 1). We then consider the graph structure shown in the left column of Figure 2 where the edge ž1 + 22 has been removed, and assume that 31 := 11. 22 == n2 and x : f o g 21 =2) + E where g(²1, 22) = [=1,~] + 22] (case 2). Interestingly, we find that the causal models in case 1 and case generate the same observed data x, which implies that there are two different causal models to interpret the same observed data. Clearly, in both the two equivalent causal structures are different, and thus 22 is unidentifiable.
of 21 and 22 on in Figure [ According
Similarly, we can also cut the edge 21 23 Figure[] [ obtaining another equiv-
alent causal graph as shown in the right of Figure 2 That is, we can have two dif- ferent 2y to interpret the same observed data, and thus 2y is unidentifiable. Since there exists many different equivalent causal structures, the latent causal vari- ables in Figure is unidentifiable. Such result is because that the effect of 21 on 22 (or 23) in latent space can be 'ab- sorbed' by the nonlinear function from z to x. We term this indeterminacy transitivity in this work. We will show how to handle this challenge in Section 4 Before that, we introduce the other two indeterminacies in latent space, which assists in understanding the proposed identifiablity result.
Later
Figure 2: Two equivalent graph structures.
3.3 SCALING INDETERMINACY IN RECOVERING LATENT CAUSAL REPRESENTATIONS
The scaling indeterminacy of latent causal variables is also an intrinsic indeterminacy in latent space. Again, for simplicity,
only consider the influence of 2 and 22 on x in Figure ! and assume that 21 == 71,, Z2 == 21 + f22 and x == f(21, z2) + E. Under this setting, if the value of z1 is scaled by 8, e.g. 8 21, we can easily obtain the same observed data x by: 1) letting 22 == _ x 21 n2 and 22) + E where 22) = [s x 21, 21]. This indeterminacy is because the scaling of the latent variables 2; can be 'absorbed' by the nonlinear function from z to x and the causal functions among the latent causal variables. Therefore, without additional information to determine the values of the latent causal variables 2,, is only possible to identify the latent causal variable up to scaling, not exactly recovering the values. In general, this scaling does not affect identifying the causal structure among the latent causal variables. We will further discuss this point in Section
3.4 PERMUTATION INDETERMINACY IN RECOVERING LATENT CAUSAL REPRESENTATIONS
Due to the nature of pll-posedness, latent causal repre- sentation learning suffers from permutation indetermi- nacy, where the recovered latent causal representations have an arbitrary permutation. For example, assume that the underlying (synthetic) latent causal represen- tations are corresponding to the size, color, shape, and location of an object, and we obtain the recovered latent causal representations 21, 22, 23, 24. Permutation indeterminacy means that we can not ensure that the recovered 21 represents which specified semantic information, e.g., the size or the color. Therefore, without additional information to determine the order of the recovered latent causal representations 2;, it is only possible to identify the latent causal representations up permutation.
21
2
Figure 3: A Causal Supergraph.
Although the flexibility and ambiguity in latent space causes permutation indeterminacy bringing some troubles, they also bring a benefit for learning latent causal representations with straightforward way to handle the causal order on latent causal representations. That is, we can always pre-define a causal order 21, 22, 23, 24, without specified semantic information for the nodes. Based on this,
4

---

Under review as a conference paper at ICLR 2023
As result, the four predefined nodes 21, 22, 23, 24 are enforced to learn four corresponding latent variables in the correct causal order. For example, assume that correct underlying causal order is: size > color > shape location. Since 21 is the first node in the predefined causal supergraph, 21 is enforced to the semantic information of the first node in the correct underlying causal order, e-g., the size. Similarly, 22 is enforced to the semantic information of the second node in the correct underlying causal order, e.g., the color. The proposed causal supergraph can naturally ensure directed acyclic graph estimation learning causal representation, avoiding DAG constraints (such as the one proposed in Zheng et al. (2018)). We will further show how implement it in the proposed method for learning latent causal representations in Section 5
further pre-define causal supergraph depicted by Figure 3
IDENTIFIABLE LATENT CAUSAL REPRESENTATIONS WITH WEIGHT-VARIANT LINEAR GAUSSIAN MODELS
As discussed in Section 3.2 the key factor that impedes identifiable causal representations is the transitivity in latent space. Note that the tran- sitivity is because the causal influences among the latent causal variables may be 'absorbed' by the nonlinear mapping from latent variables z to the observed variable x. To address this is- sue, we allow causal influences among latent causal variables be modulated by an addition- ally observed variable represented by the red edge in Figure 4 Intuitively, variant causal in- fluences among latent causal variables cannot be 'absorbed' by an invariant nonlinear map- ping from resulting in identifiable causal representations. We assume x and z satisfy the following causal models:
No Rg
R4
Castol
Figure 4: The proposed Latent Causal Model.
2: (B), (u)) 2; = (2)(2) + n4.
f(z) +
where
each noise term n; is Gaussian distributed with mean (u) and variance 31,(( both 3,1 and f3,2 can be nonlinear mappings. Moreover, the distribution of n, is modulated by the observed variable u.
In Eq. (2), >x(u) denote the vector corresponding to the causal weights from to 2;, e.g. (u)), and each AJ,5 could be a nonlinear mapping.
(u = (u)
In Eq. (3), denote nonlinear mapping, and is independent noise with probability density function (E).
In Eq. (2), assume the latent causal model of each satisfies a linear causal model with the causal weights being modulated by u, and we call it weight- linear Gaussian model. Therefore, p(z(u) satisfies the following multivariate Gaussian distribution:
P(A,)) (z) NN,
HHere causal supergraph is defined directed acyelic graph in which there is directed edge for any two variables.
5

---

Under review as a conference paper at ICLR 2023
with the mean μ and the covariance matrix E computed by the following recursion relations Nasrabadi 2006 Koller & Friedman 2009):
(Bishop
Σ
(1))y) + B,,(u), Eq Σ
(u)) (1))
14
JEPa
jEPa,
(5)
E, = Σ
AnA(( for i i
REpa;
where pa, denotes the parent nodes of 2;-
Furthermore, this multi-variate Gaussian can be re- re-formulated with the following exponential family:
((T/),,)(z/u)) = Z(,, B, u)
exp T (z)m(u))
(6)
where the natural parameter (u) - vec ( (2,,) denotes the normalizing constant, and the sufficient statistic T(z) [2; vee (2x?)) ('vec" denotes the vectorization of matrix). We further denote by Tiin the minimal sufficient statistic and by k its dimension, with
2n S k S [21, 22, .... 2n, 27. 22 .... 21 or in other words, there is no edges among the latent variables z, So with
(n (n + 1)) In particular, & 2n corresponds to the case when mz)
while k
full-connected causal graph different causal structures over z, the dimension k may vary. For the graph structure in Figure miin' (2) = [1 22; 23, 24, 21, 2 =3; =1. 21 22, 23 23 and k 11.
(n (n 1))/2 corresponds
The following theorem shows that under certain assumptions on the nonlinear mapping f and the variability of u, the latent variable can be identifiability up trivial permutation and scaling.
Theorem 4.1 Suppose latent causal variables z and the observed variable follow the generative models defined in Eq. 7 Eq. with parameters (f, (, Assume the following holds:
(i) The set {x E (x) = 0} has measure zero (i.e., has at most countable number of elements), where Ie is the characteristic function of the density Pe-
(ii) The function in Eq. B is bijective.
(iii) There exist distinct points up, U1: Ug such that the matrix
L (= m7uu).... m742)) 17(uu))
of size k is invertible.
(iv) For each i, (0) O,
then the true latent causal variables z are related to the estimated latent causal variables 2 by the following relationship:
2 == Pt + C, where P denotes the permutation matrix with scaling, denotes constant vector:
Intuitively, the identifiability result benefits from the weight-variant linear Gaussian model in Eq. 2 The auxiliary observed variable modulates the variant weights among latent causal variables. which can be regarded as an intervention and contributes to the identifiability, so a wrong causal structure over the latent variables will violate the assumed model class. As challenging problem, causal representation learning is known to be generally impossible without further assumptions. The proposed weight-variant linear Gaussian model explores the possibility of identifiable causal representation learning, without any special causal graphical structure constraints. The proof is given in Appendix.
The above theorem has shown that the latent causal variables can be identified up to trivial permutation and linear scaling. With this result, identifiability of causal structure latent space reduces to the identifiability of the causal structure in observed space. With the help of recent progress in causal discovery from heterogeneous data (Huang* et al. 2020). the following corollary shows that the causal structure among latent variables z is also identifiable up the Markov equivalence class. Moreover, although there exists scaling indeterminacy for the recovered latent variables as stated in Theorem 4.1 it does not affect the identifiability of the causal structure.

---

Under review as a conference paper at ICLR 2023
Corollary 4.2 Suppose latent causal variables and the observed variable follow the generative models defined in Eq. [1] Eq. 3 and that the conditions in Theorem 4.1 hold. Then, under the assumption that the joint distribution over z and u is Markov and faithful to G.. a the acyclic causal structure among latent variables z can be identified up to the Markov equivalence class of Gz. where Gzau denotes the causal graph z and Gz denotes the causal graph
5 LEARNING CAUSAL REPRESENTATIONS WITH WEIGHT- VARIANT LINEAR GAUSSIAN MODELS
Based on the identifiable results above, in this section, propose structural causal variational autoencoder (SuaVE) to learn latent causal representations. We first propose a structural causal model prior relating to the proposed weight-variant linear Gaussian model. We then show how to incorporate the proposed prior into the traditional VAE framework (Kingma & Welling (013a). We finally give consistency result for the proposed SuaVE.
5.1
PRIOR ON LEARNING LATENT CAUSAL REPRESENTATIONS
Since we can always have super causal graph for the causal structure among as shown in Figure as discussed in 3.4 the corresponding weight matrix À [V] (u),.... A, (u) can be constrained as upper triangular matrix with zero-value diagonal elements. As result, the causal model for the latent causal variables, e.g., Eq. | and Eq. can be reformulated with the following probabilistic model.
p(zlu) = p(21 II p(z = II N( 2,
(8)
where
Pz. Σ (e + (u),
# = 822(1).
(9)
The proposed prior naturally ensures directed acyclic graph estimation, avoiding additional DAG constraint. contrast, traditional methods, i.e. Causal (Yang al. (002),, employ a relaxed DAG constraint proposed by Zheng et al. (2018) to estimate the causal graph, which may result in cyclic graph estimation due to the inappropriate setting of the regularization hyper parameter.
5.2 SUAVE FOR LEARNING LATENT CAUSAL REPRESENTATIONS
The nature of the proposed prior in Eq. g and Eq. ggives rise to the following variational posterior:
q(2|x, u) = (22////////01 u) II ((2 2c,, u.
II NW',, 622, 1=1
(10)
where:
P/s = X. (x,,
6" = (x,( u).
(11)
Therefore, we can arrive at simple objective:
max B,((/xx.n)) (p(x/z, u)) DKL u)llp(z)u),,
(12)
where D denotes Kullback-Leible divergence. Note that the proposed VE is different from hicrarchical VAE models (Kingma et al., 2016) (Sonderby et al. 2016 Maaløe et al. [0019] Vahdat & Kautz 2020) in that the former has rigorous theoretical justification for identifiability, while the latter has no such supports without further assumptions. Besides, the theoretical justification implies the consistency of estimation for the proposed SuaVE follows:
Theorem 5.1 Assume the following holds:
(i) The variational distributions Eq. 10 contains the ture posterior distribution pr, pr, (,)xx u).
(ii) We maximize the objective function: max (pxxxz, u)) - DKL(((zzix, ui) ())
then in the limit of infinite data, the proposed SuaVE learns the true parameters = (f, X, 3) up to Pž c, where P denotes the permutation matrix with scaling, c. denotes constant vector.
7

---

Under review as a conference paper at ICLR 2023
6 EXPERIMENTS
Synthetic Data We first conduct experiments synthetic data, generated by the following process: we divide the latent noise variables into M segments, where each segment corresponds to one conditional variable as the segment label. Within each segment,
first sample the mean 1 and variance 2 from uniform distributions [-2, 2] and [0.01, 3], respectively. We then sample the weights Xi from uniform distributions [0.1, Then for each segment, we generate the latent causal samples according to the generative model in E Finally, we obtain the observed data samples x by an invertible nonlinear mapping. More details can be found in Appendix.
Comparison We compare the proposed method with identifiable VAE (iVAF) Khemakhem et al. (2020) B- VAE Higgins et al. (2017), Causal VAHaang et al. (202T), and vanilla VAH Kingmaa & Welling 2013 b0. Among them, iVAE has been proven to be identifiable so that it is able to learn the true independent noise variables with certain assumptions. While 3- VAE has no theoretical support, it has been widely used in various disentanglement tasks. Note that both methods assume that the latent variables are independent, and thus they cannot model the relationships among latent variables. To make a fair comparison in the unsupervised setting, we implement an unsupervised version of Causal which is not identifiable.
Performance metric Since the proposed method can recover the latent causal variables up to trivial permutation and linear scaling, we compute the mean of the Pearson correlation coefficient (MPC) to evaluate the performance of our proposed method. The Pearson correlation coefficient a measure of linear correlation between the true latent causal variables and the recovered latent causal variables. Note that the Pearson coefficient is suitable for iVAE, since has been shown that iVAE can also recover the latent noise variables up linear scaling under the setting where mean and variance of the latent noise variables are changed by u (Sorrenson et al.) 2020). To remove the permutation effect, following Khemakhem al. (2020), first calculate all pairs of correlation and then solve linear sum assignment problem to obtain the final results. A high correlation coefficient means that we successfully identified the true parameters and recovered the true variables, up component-wise linear transformations.
SanV SHarse
Ca Fet Cieeee
(a)
(c)
Figure 5: Performances of the proposed SuaVE in comparison to IVE,, B-VAE, VAE and Causal in recovering the latent causal variables on synthetic data with different numbers of segments (a), and with different numbers of the latent causal variables (b). The proposed Sua VE obtain the best MPC, compared with the others. (c) The SHD obtained by the proposed SuaVE and Causal VAE.
Results We compared the performance of the proposed Su to some variants of VAE mentioned above. We used the same network architecture for encoder and decoder parts in all these models. In particular, we add a sub-network to model the conditional prior in both iVVAE and the proposed SuaVE. We further assign an linear SCM sub-network to model the relations among latent causal variables in the proposed VE. We trained these 5 models on the dataset described above, with different numbers of segments and different numbers of latent causal variables. For each method, we use different random seeds for data sampling. Figure (a) shows the performance on two latent causal variables with different numbers of segment. The proposed SuaVE obtains the score 0.96 approximately for all the different numbers of segment. contrast, V-VAF, VAE and Causal fail to achieve good estimation of the true latent variables, since they are not identifiable. iVAE obtains unsatisfactory results, since its theory only holds for i.idd latent variables. Figure (b) shows the performance in recovering latent causal variables on synthetic data with different numbers of the latent causal variables. Sigure5] depicts the structural Hamming distance (SHD) of the recovered skeletons, causal graphs and completed partial directed acyelic graphs (CPDAG) by the proposed SuaVE and CausalVAE. Since CausalVAl obtains fully connected graphs, the SHD of the recovered CPDAG is the same one of the recovered skeletons.
8

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 6: Performance fMRI Data. (a) the performance of the proposed SuaVE in comparison
to iVAE, B-VAE, VAE and CausalVAE recovering the latent six signals. (b) The skeleton of the anatomical connections given in Bird & Burgess (0008) (c) The recovered skeleton by the proposed
SuaVE, where the dashed lines indicate errors.
fMRI Data Following Ghassami et al. (2018), we applied the proposed method to fMRI hippocam- pus dataset (Laumann Poldrack| 2015), which contains signals from six separate brain regions: perirhinal cortex (PRC), parahippocampal cortex (PHC), entorhinal cortex (ERC), subiculum (Sub), CA1, and CA3/Dentate Gyrus (CA3/DA) the resting states on the same person in 84 successive days. Each day is considered as different u, thus u is a 04-dimensional vector. Since we are interested recovering latent causal variables, we treat the six signals as latent causal variables by applying random nonlinear mapping on them to obtain observed data. We then apply various methods on the observed data to recover the six signals. Figure (a) shows the performance of the proposed Su in comparison iVAE, 8- VAE, VAE and Causal in recovering the latent six signals. 3-VAE aims to recover independent latent variables, and it obtains an interesting result: enforcing independence (e.g- B = 25, 50) leads worse MPC, and relaxing it (though contracting to its independence assumption) improves the result (e.g. B = 4). This is because the latent variables given the time index are independent in this dataset.
We further verify the ability of the proposed method to recover causal skeleton. We use the anatomical connections (Bird & Burgess 2008) as a reference shown in F since causal connections among the six regions should not exist if there is no anatomical connection. After recovering the latent causal variables, we estimate the causal skeleton by threshold (0.1) on the learned A.,, to remove some edges, and obtain the final causal skeleton. Figure 6 (c) shows an example of the estimated causal skeleton by the proposed SuaVE. Averaging over 5 different random seeds for the used nonlinear mapping from latent space to observed space, structural Hamming distance (SHD) obtained by the proposed SuaVE 5.0 + 0.28. In contrast, iVAE, 3- and VAE assume latent variables to be independent, and thus can not obtain the causal skeleton. We further analyse the result of VAE with 3 = 4 by using the PC algorithm (Spirtes et al.) (00I) to discovery skeleton on the recovered latent variables, and obtain SHD = 5.8 = 0.91. This means that: 1) 3-VAE with B = does not ensure the strong independence among the recover latent variables as it is expected, 2) the proposed Sua VE is effective one-step method to discovery the skeleton. The unsupervised version of Causal is not identifiable, and can also not ensure the relations among the recover variables to be causal skeleton in principle. In experiments we found that CausalVAE always obtains fully-connected graphs for the 5 different random seeds, for which SHD is 9.0 + O.
CONCLUSION
Identifying latent causal representations is known be generally impossible without certain assump- tions. Motivated by recent progress in the identifiability result of nonlinear ICA, we analyse three intrinsic indeterminacies in latent space, e.g., transitivity, permutation indeterminacy and scaling indeterminacy, which provide deep insights for identifying latent causal representations. To address the unidentifiable issue due to transitivity, we explore a new line of research the identifiability of latent causal representations, that of allowing the causal influences among the latent causal variables to change by an additionally observed variable. We show that latent causal representations, modelled by weight-variant linear Gaussian models, can be identified up to permutation and scaling, with mild assumptions. We then propose a novel method to consistently learn latent causal representations. Experiments synthetic and fMRI data demonstrate the identifiability and consistency results and the efficacy of the proposed method. Future work may include developing effective methods for learning latent causal structures and exploring nonlinear causal representations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the non gaussian and heterogeneous cases. NeurIPS, 2021.
Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In ICML, pp. 249-257, 2013.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 1788)1198--1888, 2013.
Chris M Bird and Neil Burgess. The hippocampus and memory: insights from spatial processing. Nature Reviews Neuroscience, 93):1122-194, 2008.
Christopher M Bishop and Nasser M Nasrabadi. Puttern recognition and machine learning, volume 4, Springer, 2006.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representa- tion learning. arXiv preprint arXiv:2203. 16437, 2022.
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. In NeurlPS, 2019. 2019.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in vaes. In NeurlPS, 2018.
Benjamin Frot, Preetam Nandy, and Marloes H Maathuis. Robust causal structure learning with some hidden variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 81 (3):459-487, 2019.
AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domain causal structure learning in linear systems, Advances in neural information processing systems, 31, 2018.
I. Higgins, Loie Matthey, A. Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
B. Huang*, K. Zhang* J. Zhang. J. Ramsey, R. Sanchez-Romero, C. Glymour, and B. Schölkopf. Causal discovery from bederogeneous/noostttiionnng data. JMLR, (89), 2020.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In AISTAS, pp. 2207-2217. PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML, 2018.
D. P. Kingma and M Welling, Auto-encoding variational bayes. NewrlPS, 2013a.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312v61312.61 14, 2013b.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In NeurlPS, volume 29, pp. 4753-4751, 2016.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. NeurIPS, 2021.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
10

---

Under review as a conference paper at ICLR 2023
Sébastien Lachapelle, Pau Rodriguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien, Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. arXiv preprint arXhv:2107.10099 2021.
Timothy O. Laumann and Russell A. Poldrack, 2015.
URL https://openfrri.orm
dataset/ds000030
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. 13557-13603. PMLR, 2022.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In ICML, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In ICMI., 2020.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: very deep hierarchy of latent variables for generative modeling. In NeurIPS, 2019.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, 2000.
Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. JMLR, (5688)2009920533 2014.
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)6612-63) 2021.
Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärimen. Estimation linear non-gaussian acyclic models for latent factors. Neurocomputing, 72(7-9):2024-200 2009.
Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. JMLR, 7(2), 2006.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NeurlPS, 2016.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-foww networks (gin). arXiv preprint arXiv:2001. 04487,, 2020.
P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, Cambridge, MA, 2nd edition, 2001.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In ICMI, pp. 10401-10412, PMLR, 2021.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. In NeurIPS, 2020.
Julius Von Kügelgenn Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in neural information processing systems, 2021.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In NeurIPS, 2020.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He. Zhi Geng, and Kun Zhang. Identification of linear non non-gaussian latent hicrarchical structure. In International Conference on Machine Learning. pp. 24370-24387. PMLR, 2022.
11

---

Under review as a conference paper at ICLR 2023
Mengyue Yang. Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. In CVPR, 2021.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint 1rX0::210 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Learning latent causal dynamics. arXiv preprint arXiv:2202.04828, 2022.
Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint
2012.
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In NeurlPS, 2018.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 THE PROOF OF THEOREM
The proof of Theorem 4. done in three steps. The steps is the same nonlinear Khemakhen et al. 2222)) which builds a connection between any two different solutions by using a simple convolutional trick made possible by assumption (i). The step II consists of building a linear connection between any two different solutions by assumptions (iv). This step is different with nonlinear ICA Khemakhem et al. (2020) in that we employ the assumptions of multivariate Gaussian as Eq. 6 which can model the linear causal relations among the latent causal variables, while nonlinear ICA Khemakhem et al. (2020) assumes independent noise variables. The step III shows that we can recover latent causal variables up to a linear transformation by the identity theorem for polynomials and assumption (iii). The step IV shows that the linear transformation can be reduced to permutation transformation by assumption (iv).
Step I: Suppose we have two sets of parameters (f, 9) and ô (f, T. ( such that (x = P(F, (x/u) for all pairs (x, u). Then:
I Prtxsseelu/ppoo(x/)dd
/ P(t.x.a)z/ (x/))
(13)
/ ( f(z) dz / ((t f(z)) dz
(14)
(TTA (f)) (x)p=(x- x) (0
x) dx
(15)
/ Prosata(R)//xx3165 / P(T.S.B....) (x)pc(x- (1)) 8)
(16)
M 4
(FTTA * (x) = (fT.A....P.u) * (x))
(17)
FiTTAAEN)W)) (w) F. (w)
(18)
F(iIAA (
(19)
4
PTA.E (x) = -((..A..f..,xx))
(20)
where:
in Eq. 15| J denotes the Jacobian, and we introduce here the volume of matrix denoted vol.
as the product of the singular values of J. When J is full column rank, vol. = vdet J and when J is invertible, volJ |det J/. We made the change of variable f(z) on the left hand side, and fzz) on the right hand side.
in Eq. 16
introduced:
PTT.A.B.f.,) (x) = P(T,A,3) (f (x) [)voJJJ (x)1x((x),
(21)
on the left hand side, and similarly on the right hand side.
in Eq.
used for the convolution operator.
in Eq.
18
used F[.. to designate the Fourier transform.
. in Eq.
19
we dropped e from both sides as it is non-zero almost everywhere (by assumption
Step II By taking the logarithm both sides of Eq. 20 and replacing P(T,A,3) by its expression from
Eq. we have:
log volJp- + TT (x))m(u) - llg 2(A,3,u)
(22)
log volJ,-1(()) + T (x))) log 2
(23)
13

---

Under review as a conference paper at ICLR 2023
Let Ug. U1 ....... 4k+1 be the points provided by assumption (iii) of the Theorem. Similar to nonlinear
ICA Khemakhem et al. (2020), we define 17(u) = 77(1) (7(u)). We plug each of those u, | = 1..., k in Eq. 23 obtain k equations. For example, for ug and Us we have two equations:
equations. We subtract the first equation for u, from the remaining
vol.// 1 (x) + T (f (x)) log ( B, uo) =
(24)
log vol.J= (x) T (f (x)))((u)) log 2 3, un).
and
log vol. (x) + T (x))m(u log 7 (2, up) - log 2(2, ub),
(25)
log vol. (x) + T (
Using Eq.
25
subtracts Eq. 24 removing the terms that do not include u. we have:
T (f (x)) log 7(>, u2) u.)) TT (f + log Z(A, B, uo)
(26)
-1 (f- (x)) log 2 uz) T (x)) + log ((, B, uo).
As result, arrive at:
3(,,B,ub) u.)
(f (x)), i)))) log 2(X, B, up) 2(,, w.)
(T (x)-)(x)), m(,:)) log
(27)
Let L be the matrix defined in Eq. 7 in assumption (iii), and É similarly (L is not necessarily invertible). Define bj log 2 un ((A, and b the vector for by Expressing Eq. 27 for all
2(A.33,u (
points uy in matrix form, we get:
L L'
f +b
(28)
We multiply both sides of Eq.
28 by the inverse of L to find:
T(f T(f A (x)) + c,
(29)
where (LT)- and (L7)-'b.
Step III As mentioned in Eq.
6 the sufficient statistic T(z) [z; vec(zz") In this case, the
relationship Eq. 29 becomes:
z² A
2²
+ c,
(30)
Zi.j
Zi.j
where z denotes [e],..., 2.). x² denotes [2?,..., 2111 Zi,j denotes the vector whose elements are 2;2;
for all * j. A in block matrix can be rewritten as:
A (1)
A (2)
A
A =
A (4)
A (5)
A
(31)
A (7)
A (8)
A (9)
and c as:
c(l)
c=
((2)
(32)
((3)
Then, we have:
z = A (1) + A (2) + A (3)
21. ( +
( (1)
(33)
z² = A (1) + A (5) A (6) 21 + (2))
(34)
So we can write for each 2₁:
2 Σ (1) + Σ (2(2
(3) [A 2:2 + (1)
+
(35)
x = Σ (A)))))
2)) + Σ (4()))) + Σ
A.j 242 + ( (2)
(36)
3
14

---

Under review as a conference paper at ICLR 2023
Squaring Eq. 35
we have:
2
Σ
21)))))
+
Σ jihi
(3) (
211)) +(c/))2+
(37)
(b)
(c)
is notable that Eq. 36 and Eq. 37 are derived from Eq. which holds for arbitrary x. then by the
assumption (ii) in Theorem 4.1 that f is bijective mapping from z to and thus Eq. 36 and Eq. 37
must holds for z everywhere. Then by the fact that the right sides of the Eq. [3 and Eq. 37 are both
polynomials with finite degree, we have each coefficients of the two polynomials must be equal. In more detail, for the term (a) in Eq. 37
(AA E(4 + 4
(38)
Compared with Eq. 36
since there is no term 2f in Eq.
36
we must have that:
A (2) O.
(39)
For the term (c) in Eq. 37
(A
(2(2))) +
(40)
jpft
jtt
Compared with Eq.
36
since there is no term
(2:2,)² 2 in Eq.
36 we must have that:
A (3)
(41)
As a result, Eq.
33 becomes:
z A (1) c!))
(42)
The above equation indicates the latent causal variables can be recovered up to linear transformation.
Step IV We then show that the linear transformation matrix A (1) in Eq. 42 must be a permutation matrix. Since our assumptions in Theorem 4. includes the assumptions for identifying the Gaus- sian noise variables n,, the Gaussian noise variables n. can be recovered up to linear scaling and permutation according Sorrenson al. (2220):
n Pn Cz
(43)
where P denote the permutation matrix. With Eq. 43 the Eq. 42 can be rewritten as follows:
B(P + c,)) A (1) (Bn) (1))
(44)
[BP - A ()) = (1) Bc,, + -Bar
Bc.,
(45)
where B and Ô denote lower triangular matrix, for which the main diagonal are 1, the others depend on (u) and (u), respectively. By differentiating Eq. 45 with respect to â, we have:
(BP- - A AOBII
(46)
where denote the identity matrix, which implies that:
BP = A
(47)
Then (1) must be an identity matrix. To understand this point, we give a 2-dimensional case as follows, which can be casily extended to n-dimensional case. Without loss of generality, we assume P to be diagonal with elements $1,1, $2,2-
(
81,1 (1,1112(u) (s)))
0 )
(1)
(Arltw) 1) =
A + A (1) (
A
(1)
$2,2
A
2,2
A (1) (1) (
+ 2,2
22
(1)
(48)
Then we have:
82,2, and 1 (0) = 0). The proof of other case for P is similar to the above procedure.
1
0,
$1,1,
A
(by assumption (iv)
15

---

Under review as a conference paper at ICLR 2023
A.2 UNDERSTANDING ASSUMPTIONS IN THEOREM 4.1
Gaussian assumption the latent noise variables Our first assumption is implicitly enforcing Gaussian distribution on the latent noise variables. Note that the assumptions of nonlinear ICA (Keemahhem al. 2020 on the noise could be broad exponential family distribution, e.g., Gaussian distributions, Laplace, Gamma distribution and so on. This work consider Gaussian distribution, mainly because it can be straightforwardly implemented for the re- re-parameterization trick in VAE. 2) it is convenient for simplifying proof process (As shown in Step III in the proof.). It is worthwhile and promising to extend the Gaussian assumption to exponential family for future work.
Linear model assumption for the latent causal variables As the first work to discuss the relation between the change of causal influences and identifiability pf latent causal model, this work simply consider linear models for the latent causal variables, because it can be directly parameterized, which helps us to analyse the challenge of identifiability and how to handle it. In addition, together with Gaussian noise, we arrive the same probabilistic model linear Gaussian Bayesian networks, which a well-studied model as mentioned in Eq. [ We expect that the proposed weights- varriatt linear models can motivate more general functional class in the future work, e.g., nonlinear additive noise models.
Changes of weight and Assumption (iv) We allow changes of weight (e.g., causal influences) among latent causal variables across u to handle the transitivity problem. We argue that this is not necessary condition for identifiability (Existing methods, including sparse graph structure and temporal information. could be regarded as feasible ways to handle the transitivity.), but it is sufficient condition handle the transitivity, and provide new research line for causal representation learning. Assumption (iv) is to handle special case as follows: (u,(U)) = X, + b where b is non-zero constant. For example, if 22 = (X1 2 + b) + f2, then the term b21 is still unchanged across all u, and thus can still be 'absorbed' by the nonlinear mapping from latent variables z to the observed variable x due to the transitivity, which results in non-identifiabiiity. Note that assumption (iv) is sufficient to handle the special case, but it may not be necessary. There may exist possible assumption to relax the assumption (iv).
Assumptions (i)-(ii)) All three assumptions are motivated by the nonlinear ICA literature (hee makhem et al. 2020 which is to provide guarantee that recover latent causal variables 2; up to a linear transformation. Note that three assumptions also provide a support for recovering the latent noise variables up linear scaling and permutation.
A.3 THE PROOF OF COROLLARY 4
Theorem 4.1 has shown that the latent causal variables z can be identified up to trivial permutation and linear scaling. Hence, the identifiability of the causal structure in latent space can be reduced to the identifiability of the causal structure in observed space. Moreover, we need to show that the linear scaling does not affect theoretical identifiability of the causal structure.
For the identifiability of the causal structure in observed space from heterogeneous data, fortunately, we can leverage the results from Huang et al. (020)). Corollary 4.2 relies on the Markov condition and faithfulness assumption and the assumption that the latent change factor (i.e., causal strength in the linear case) can represented as function of the domain index u. Hence, it relies on the same assumptions as that in Huang* R et al. (2020).
has been shown Huang* et al. (020)) that if the joint distribution over z and u (z and are observed variables here) are Markov and faithful to the augmented graph, then the causal structure over z U u can be identified up to the Markov equivalence class, by making use of the conditional independence relationships.
Next, we show that the Markov equivalence class over z also identifiable. Denote by M the Markoy equivalence class over U and by M, the Markov equivalence class over z. Then after removing variable u in M and its edges, the resulting graph (denoted by M³) is the same as Mg. This is because of the following reasons. First, it is obvious that My and M, have the same skeleton. Second, in
16

---

Under review as a conference paper at ICLR 2023
this paper, has an edge to every 2; when considering as latent noise variables, because all causal strength and noise distributions change with u. Hence, there is no v-structure over u, 24, and 2j, so is not possible to have more oriented edges in My. Therefore, Mr and M, have the same skeleton and the same directions.
Moreover, the conditional independence relationships will not be affected by linear scaling of the variables, so the conditional independence relationships still hold in the identified latent variables. Furthermore, for Gaussinn models, independence equivalence is the same distributional equivalence. This is because of the following reasons. First, since we are concerned with linear Gaussian models over the latent variables, the identifiability up to equivalence class also holds for score-basee based methods that use the likelihood of data as objective functions. This is because independence equivalence, i.e., two DAGs have the same conditional independence relations, is the same as distributional equivalence for iinusr-Gaussian models, i.e., two Bayesian networks corresponding to the two DAGs can define the same probability distribution. That is to say, score- based methods also find the structure based independence relations implicitly. Because the scaling does not change the independence relations, it will also not affect the identifiability of the graph structure.
Therefore, the causal structure among latent variables z can be identified up to the Markov equivalence class.
A.4 THE PROOF OF THEOREM 5.1
The proof of Theorem 5.1 is similar to the proof of Theorem 4 in non-linear ICA Khemakhem et al. (2020). If the variational posterior q(z)x x, u) is large enough to include the true posterior p( u), then by optimizing the loss, the KL term will be zero, and the loss will be equal to the log-likce ligeiiel Since our identifiability is guaranteed up to permutation and scaling, the consistency of maximum log- ligliikl means that we converge to the true latent causal variables up to permutation and scaling classes, in the limit of infinite data.
A.5 IMPLEMENTATION DETAILS
A.5.1 EXPERIMENTS FOR FIGURE 5 (A)
Data For experimental results Figure (a) the number of segments (e.g..u) M is 10,20,30.40
and 50 respectively. For each segment, the number of latent causal variables is 2 and the sample size 1000. We consider the following structural causal model
R1 == (((((u)
n2 == N(8(u))..
(49)
21 711
22 21,2 (1)21 n22
(50)
where we sample the mean (()),. and variance 3(u),,2 from uniform distributions [-2, 2] and [0.01,3] 3), respectively. We sample the weights X1.2 (u) from uniform distributions [0.1, 2]. We sample latent variable according to Eqs. 49 and 50 and then mix them using 2-layer multi-layer perceptron (MLP) to generate observed x.
Network and Optimization For all methods, we used encoder, e.g. 3-layer fully connected network with 30 hidden nodes and Leaky-ReLU activation functions, and decoder, e.g. 3-layer fully connected network with 30 hidden nodes and Leaky-ReLU activation functions. We also use 3-layer fully connected network with 30 hidden nodes and Leaky- ReLU activation functions. For optimization, use Adam optimizer with learning rate Ir Le 3. For parameerr,,,,ss, we set for the V-E. For VAF, we set the
as suggested in the paper Yang
al. 2021
A.5.2
EXPERIMENTS FOR FIGURE 5 (B) AND (C)
Data For experimental results of Figure (b) and (c), the number of segments is 30 and sample
size is 1000, while the number (e.g., dimension) of latent causal variables is 2,3,4,5 respectively. For
17

---

Under review as a conference paper at ICLR 2023
2- 2-dimensional latent causal variables, the setting is the same experiments for Figure (a). For 3-dimensional case, we consider the following structural causal model:
n1 (1 n2 (0000)),,,()22.200
ng ((
(51)
21 = 11,,
22 == ( + n2,
23 = 22,3( +1.
(52)
For 4-dimensional case, we consider the following structural causal model:
n1 = ((3
n2 = ((8(U)).1
(53)
ng ((3(1)),1,8((1))))
n4 N (1(
(54)
21 = 711,
= (1,2 112
(55)
23 = 2 ( 22 13:
24 (2)4 n14
(56)
For 5-dimensional case, we consider the following structural causal model:
n1 ,(
n2 ((B(u)).1,B(U)22)
(57)
f1g = ((8(()).1,8(0)) (()),2),
14 = (,,
= (5==N($(u),,-B(()bba)
(58)
21 = T
22 (1,2( 112
(59)
:= 22,3 22 + 133
24 = (2)2 +24
25 = ) +
(60)
sample the mean B(u);, and variance B(u);,2 from uniform distri- butions 2, and [0.01. 3/, respectively. We sample the weights {u,((u) from uniform distributions
Again, for all these Eqs 51
[0.1.2]. We sample latent variable according these equations, and then mix them using a 2-layer multi-layer perceptron (MLP) to generate observed x.
Network and Optimization The network architecture, optimization and Syper-parameten are the same used for experiments of Figure 5 (a), except for the 5-dimensional case where we use 40 hidden nodes for each linear layer.
A.6 EXPERIMENTS ON CHANGES OF PART OF WEIGHTS
In this part, conduct experiments changes of part of weighs. To this end, we consider two cases on the following graph: + + (data details as mentioned in section A.5.2 for 3-dimensional case): Change 1: the weight on the edge of 21 22 changes across u, while the weight on the edge of 22 + 23 remains unchanged across u, Change 2: the weight on the edge of 21 + 22 changes only for the first 15 segments (e.g., u), the weight on the edge of 22 - 23 changes only for the last 155 segments (totally 30 segments). Again, the network architecture, optimization and dyper-parameters are the same as used for experiments of Figure 5 (a). The Figure shows the MPC
1.00
0.8
0.96
MBC
0.94
090
0.8
Tatent Causal Veriecee
Lnfent Cauisll Vartablle
(a)
(b)
Figure 7: Performances of the proposed SuaVE in recovering the latent causal variables on Change / (a), and Change 2 (b).
between the recovered latent variables and the true ones by the proposed SuaVE For Change 1, we can see that only the recovered z2 obtains satisfactory MPC. For Change 2, all the recovered latent causal variables obtain high MPC, which seems to be promising to obtain identifiability. To further understand this interesting result, we also use a threshold to remove some edges to obtain the final graph. Since no identifiability is provided, we measure the SHD of CPDAG. We found that for Change 2, the proposed method obtains
investigate the recovered graph in the case of Change 2. Again,
18

---

Under review as a conference paper at ICLR 2023
SHD 1.0 = 0.39. Compared to that, reported SHD 0.6 = 0.35 in Figure where the weights always change across u. In summary, it seems to be promising to obtain identifiability up to Markov equivalence class for the setting of Change 2. It would be interesting to provide theoretical insights for this case in the future work.
A.7 EXPERIMENTS ON I.I.D
As we mentioned in the main paper, our work is closely related to nonlinear ICA, and our assumptions includes the assumptions for identifying the Gaussian noise variables. To verify this point, we conduct experiments on the case where any two latent causal variables 2; are independent given u. Data details is the same mentioned in section A.5.2 but we here enforce (u) 0 for each i, j. Again, the network architecture, optimization and hyper-parameters are the same as section AA.5.2
except for learning rate (here we set Ir le 2). Figure X shows the performances of the proposed SuaVE and iVAE. We can see that iVAE is slightly better than the proposed SuaVE, both obtain satisfying results in terms of MPC.
1.000
0.950
0.925
0.875
0.850
0.825
0.800
Number Latent Causal Variables
(a)
Figure 8: Performances of the proposed SuaVE and iVAE in recovering the independent latent causal variables conditional u.
A.8 DETAILED RESULTS ON FMRI
Figure 9 shows the recovered latent six signals (Blue) and the true ones (Red) within one day by the proposed SuaVE.
The recovered latent fMRI variables within one day by iVAE are depicted by Figure I0
The recovered latent fMRI variables within one day by VAE are depicted by Figure II
The recovered latent fMRI variables within one day by 3-VAE are depicted by Figure
The recovered latent fMRI variables within one day by CausalVAE are depicted by Figure 13
Implementation Details: For fMRI data, again, we used the same network architecture for encoder (e.g. 3-layer fully connected network with 30 hidden nodes for each layer) and decoder (e.g. 3-layer fully connected network with 30 hidden nodes for each layer) parts in all these models. For prior model in the proposed SuaV VE and IVAE, we use 3-layer fully connected network with 30 hidden nodes for each layer. We assign an 3-layer fully connected network with 30 nodes to generate the weights to model the relations among latent causal variables the proposed SuaVE. For setting as recommended by the authors YYang et al. 2021), for generating observed x, we use an invertible 2-layer multi-layer perceptron the fMRI with different 5 random seeds.
we
set 3 = 4, 25, 50 for 3- V-V.E. For CausalVAE, we the hyper-parameee
19

---

Under review as a conference paper at ICLR 2023
Mmwa
Figure 9: Recovered latent six signals (Blue) and the true ones (Red) within one day by the proposed SuaVE.
A
W
Figure 10: Recovered latent six signals (Blue) and the true ones (Red) within one day by iVAE.
20

---

Under review as a conference paper at ICLR 2023
Conficient 3.50
Mussrinn
24
Figure 11: Recovered latent six signals (Blue) and the true ones (Red) within one day by VAE.
wmm mamammn
Figure 12: Recovered latent six signals (Blue) and the true ones (Red) within one day by 3 V-V.
21

---

Under review as a conference paper at ICLR 2023
CAS Coofhoure
Figure 13: Recovered latent six signals (Blue) and the true ones (Red) within one day by CausalVAR. VAE.
22


Reviews:
[{"review_id": "woreDJ_q_HK", "reviewer": "Reviewer_FaeD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a solution to the problem of causal representation learning, which is about identifying latent variables and how they are causally related by observing only a nonlinear transformation of them. They propose a model in which the latent causal graph is linear gaussian and an auxiliary variable u is observed and has an influence on both the distribution of the gaussian noise and the coefficients of the linear models. They provide conditions under which this model is identifiable up to permutation and rescaling of the latent factors. Motivated by this result, they provide a method based on a simple variational lower bound to estimate the mixing functions. Compelling experimental results validates that the method can indeed identify the latent factors.", "summary_of_the_review": "Although the identifiability result introduced in this work appears to be novel and very interesting, I cannot recommend acceptance due to (i) Imprecision/wrongness of certain claims, (ii) lack of clarity and (iii) the lack of details regarding some experiments. I strongly encourage the authors to improve the writing of this work, since I believe the main theoretical result would be of interest to the community.", "strength_and_weaknesses": "Strengths:\n- I follow rather closely this literature and I found the main theoretical contribution (Theorem 4.1) of this work to be novel and quite interesting. I believe other ICLR participants will be interested by this result. \n\nWeaknesses:\n- Many claims are either imprecise or wrong (see below).\n- Some sections (especially Section 3.2) lack clarity (see below).\n- Thm 4.1 should be contrasted with CausalVAE further, which shares many similarities with this work.\n- Many experimental details are missing: For example, what is the number of latent causal variables used in Figure 5a? What is the number of segments available in Figure 5b? How’s the causal graph extracted from the SuaVE’s parameters to produce Figure 5c? This can be a problem for reproducibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "v7VWLSPYvgO", "reviewer": "Reviewer_2uCg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a theory for identifying latent causal structure using observational data when the weight-variant linear gaussian model assumption holds.", "summary_of_the_review": "Overall, I think this paper is very clearly written, and the quality is good. Also, it contains novel theories on the problem. I think the paper can be improved by if more preliminaries and detailed example of causal representation learning are provided.\n", "strength_and_weaknesses": "## Strength\n1. This paper is very well and claerly written.\n2. A detailed and interesting motivation in Section 3 helps a lot in understanding the problem.\n3. This paper contains extensive simulation results.\n\n\n## Weakness\n1. More background information for understanding the paper (e.g., what’s nonlinear ICA?) could help.\n2. More detailed example in the problem of causal representation learning (e.g., why do we want to learn the relations between variables Zi?) is needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hLGepgSI6j", "reviewer": "Reviewer_4oGs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In causal discovery, one starts with input variables and then learns a causal graph describing the relationships between the input variables. In recent years, a complementary question, which asks \"how to learn causal variables from low-level raw data (e.g., images)?\" has gained attention and this task is often termed \"causal representation learning\". Many recent works in the area either leverage simplistic assumptions on the underlying causal DAG (e.g., independence of latents conditional on observed auxiliary variable in iVAE). Other works leverage temporal data to deal with more general causal DAGs (e.g., CITRIS). In this work, the authors tackle the question without making assumptions on the structure of the underlying causal DAG. To tackle the question, the authors assume a linear structural equation model that governs interactions between the latents and also require to observe an auxiliary variable that impacts the weights of the weights of the structural equation model. Under the above assumptions, the authors extend the theory from iVAE to achieve identification in the proposed setting. Having shown identification of the latents the authors show that the causal DAG of the latents can be identified up to the Markov equivalence class. Lastly, authors propose an extension of the iVAE based procedure by adapting the prior based on the linear SEM that is assumed. The authors show that the procedure is effective in synthetic and semi-synthetic experiments.", "summary_of_the_review": "The work proposes a new DGP to tackle the question of causal representation learning. The DGP proposed seems to be quite contrived and that is one major limitation. The proof of the theorems are correct, but the proof of the corollary is incomplete or has details missing. ", "strength_and_weaknesses": "**Strengths** \nThe authors study a very important problem and propose an interesting approach that extends works such as causal VAE. A main limitation of causal VAE is that the each latent is independent of the rest conditional on one component of the auxiliary information vector. To address this limitation, authors propose a new data generation model, the weight variant linear causal model. The weight variations in the linear causal model allow the authors to achieve identification. These weight variations are connected to interventions, which makes the setup interesting. \n\n**Weaknesses** \n\nI have several concerns that I highlight below. \n\n1. **Regarding the weight variant causal model** The new data generation process considered by the authors seems quite artificial to be honest. Having access to an auxiliary variable that impacts all weights in a linear SEM seems too much of a stretch. Do authors have some good examples that show otherwise? If yes, then authors should provide a convincing case of such examples during the rebuttal. My understanding tells me that data generation process seems to have been reverse engineered to fit the requirements of the proof. In other words, DGP has been engineered to allow for structural dependency but at the same time leverage as much of iVAE as possible.\n\n2. **Regarding Corollary 4.2** The paper leverages the work of Huang et al. to achieve identification of causal DAG up to the Markov equivalence class. Since this is quite important to the paper the authors should have fully explained if the assumptions that are required in Huang et al. are fully met. Also, the authors should have described the exact steps of the procedure to go from Markov equivalence of z and u to z only. If this was excluded due to space constraints, I am a little surprised as this seems quite important. I also do not see a discussion in depth on this in the Appendix. \nRegarding the proof of Corollary 4.2. The proof of the corollary is incomplete. The authors talk about recovery of MEC of z union u. That can be done from good old works of Pearl and Verma that date back to 1990. How to go from there to the MEC of z? \n\n\n3. **Regarding experiments** In the set of experiments that are carried out on FMRI data. I do not see a description of what is the auxiliary information. In the absence of auxiliary information how does the framework of authors even kick in. Further, the authors seem to learn the DAG just by thresholding the weights learned. I was expecting to see a set of DAGs that are Markov equivalent and some of them would correspond to the true one. \nAlso, to validate Corollary 4.2 the authors should have carried out synthetic data based experiments to show that indeed the MEC is recovered. \n\n4. **Role of weight variation model** Since weight variation is the key part of the paper. Some ablation experiments that explain how different choices for weight variation model help in different types of identification would have been useful. In the current form the model carries out global interventions and that is not realistic. What would happen if the interventions were localized, i.e., some components of u only interact with some edges, do we get partial identification at least?  \n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a_WrdFg5rV", "reviewer": "Reviewer_kexo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed SuaVE, a new identifiable model for causal representations learning. SuaVE is an extension of i-VAE model in the sense that: 1, an additional latent causal variables layer is generated conditioned on the latent noise variables;  2, the latent causal variables are modelled via linear Gaussian models with $u$-dependent causal weights. Under certain conditions, the authors shows that 1), the latent causal variables can be identified up to permutation and linear scaling; 2), the causal graph among latent causal variables can be identified up to its Markov equivalence class. The authors then proposed practical algorithms for identifying the latent causal variables and conducted both synthetic and real-world experiments. ", "summary_of_the_review": "This is an interesting paper that addresses an important question in causal representation learning; and seems to possess enough novelty to the best of my knowledge. However, due to my certain technical concerns, I can only recommend a weak accept.", "strength_and_weaknesses": "Utilizing recent advances in nonlinear ICA identifiability to propose new causal representation learning algorithms is an interesting point of view. This paper is very well-written and I expect the results can make novel contribution to the field. I also like the way that the authors explain the non-identifiability of latent causal representations, motivates how to tackle them, and avoid the constraint-based optimization of no-tears by pre-defined causal orders. \n\nCompared to other reviews, I might be less worried about the empirical evaluations.  Based on my own understanding the technical details, all the proofs seems to hold except Corollary 4.2. \n\n- The main issue is that I am not convinced how Corollary 4.2 can justify the identification algorithm proposed by the paper. To be more specific, Corollary 4.2 claims that the linear scaling of latent variables does not affect the identifiability of the latent causal structure. This only true if the identification is based on e.g. conditional independence relationships. However, this is not what the learning algorithm proposes. As I understand, SuaVE implicitly models the adjacency matrix via continuous causal weights + thresholding pruning, which I guess is inspired by the parameterization used in No-tears and many other causal discovery papers. This is fine when the dataset is fully observed. However, in the case of SuaVE, $z$ can be only recovered up to linear scaling. Therefore, the scaling of $\\lambda(u)$ *will* get affected by linear scaling of $z$. This is especially the case since the authors also uses a non-zero thresholding (0.1) to round up the weights; which essentially means that this threshold also needs to be scaled proportionally to the linear scaling of $z$, which is very tricky and not considered by the paper. Also, the $u$-dependent parameterization for causal weights $\\lambda(u)$ introduces additional issues since the existence of the corresponding edges might depend on the values of $u$ as well, which is not dealt with in Huang et al. 2020. I guess the above issue might be avoided by an alternative parameterization, in which you could decouple the parameterization of causal graphs and the edge functions (as in SCMs), where the causal graph can be modeled as via binary variables, that can be used to 0-1 mask the functions of its parent nodes.\n\n- The other concern is I am not sure how to justify the modelling choice of SuaVE (Figure 4) and how reasonable this assumption is in the real-world causal representation learning applications. To me it seems a bit unintuitive to treat the latent variables in i-VAEs as non-exogenous noises here ($n$); also the fact that the latent causal graph might vary across data points seems like a non-necessary complication. Could the authors provide some examples to justify those assumptions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MPzDnV_TZMx", "reviewer": "Reviewer_P5md", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Observed signals are often generated from latent variables with significant structure that can provide significant understanding into the dynamics of the phenomenon under investigation. This structure is often unidentifiable without strong prior assumptions. The authors study a particular model, in which noise is Gaussian and associations between latent variables are linear, and show that some aspects of the latent structure are identifiable up to some transformations.", "summary_of_the_review": "Well written paper with potentially some interesting results on latent model identifiability. The assumed structure however is quite restrictive and it is not clear what the effect of these assumptions are for performance in practice. Since this method is presented as a contribution to the causal literature I don't think it is likely to have a strong impact, simply because it relies on too many unverifiable assumptions not only on the causal structure of the underlying system but also on the functional dependencies and distributions of all variables.", "strength_and_weaknesses": "**Strengths**\n- The paper is clearly written. The authors give a lot of details into why latent structure is not identifiable which serves to motivate the proposed approach. Their model is general enough that it can be a plausible explanation for latent structure in practice.\n\n**Questions / comments**\n- It is not clear what the authors are identifying in Corollary 4.2. When marginalizing over latent variables, the structure is a mixed graph and not a DAG, and thus the equivalence class is a PAG. Moreover with the assumed model in Sec. 4, all variables Z are dependent so the distribution of the data really doesn't narrow the class of compatible models.\n- The graph in Fig. 4 is non-standard, if z_i is a function of u, then u should have an arrow into z_i. Is there any difference between Eq. (2) and $z_i := g(\\mathbf u,\\mathbf z) + n_i$?\n- Thm. 4.1 depends on a lot of unverifiable, and one may argue, unrealistic assumptions about the latent structure. I understand that these may be necessary from a theoretical standpoint but also makes this method inapplicable in practice. At the very least one needs sensitivity analysis to understand whether this method can be used in practice. The current experiments are probably insufficient to understand the proposed approach.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HzZ9PE0oI8", "reviewer": "Reviewer_DXAx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors extend the nonlinear ICA results on identifiable latent factors to when the latent factors have a causal structure. ", "summary_of_the_review": "Thank you for your submission. I have some major concerns below. I would be happy to update my score based on the authors' clarifications/ addressing of these issues. \n\n1. The discussion in the first paragraph of Section 3.1 is inconsistent with the causal graph given in Figure 1: It is extremely unlikely that n_i's are independent if they are all caused by the same variable u. This assumption that n_i's are independent seems to be necessary to make the nonlinear ICA framework applicable here so this issue should be addressed by the authors. \n\n2. The result inherits the proof of Khemakhem et al. Therefore it inherits some of the the strict assumptions of their theorem. For example, the proof, when one looks closely, assumes that for any two causal models, the noise distributions have to be identical. This is quite a strong assumption and needs to be mentioned explicitly. I recommend the authors explicitly list the set of assumptions including this one in their manuscript. \n\n3. The assumption that U is a variable that is observed, and it can be used to \"intervene\" on every edge, i.e., every configuration of latent variables and also several times as required by the theorem might be a strong assumption. It might be better to dissect this assumption a bit more to justify when it can be achieved in practice.\n\n\nMinor Comments and Suggestions: \n\nThe main contribution is in Section 4. Although Section 3 gives a nice exposition with examples, the observations here are not surprising that the mapping can absorb such indeterminacies. \n\n\"causal supergraph\"\nI don't believe there is a need for a new name here. This fully connected DAG is known as a tournament in graph theory. Also known as a total order.\n\n\"To address this issue, we allow causal influences among latent\ncausal variables to be modulated by an additionally observed variable\"\nthe word \"modulated\" is not well-defined here.", "strength_and_weaknesses": "There are certain inconsistencies in the assumptions that need to be addressed by the authors. \n\nThe assumptions might be a bit too strong/unrealistic. This is evidenced by the fact that the equivalence class reduces to a single graph given the specified assumptions. If the authors showed that these assumptions are also necessary, I think that would be an interesting contribution. But I don't see such an argument. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject