Identifiable Latent Polynomial Causal Models through the Lens of Change
Authors: Yuhang Liu
Zhen Zhang
Dong Gong
Mingming Gong
Biwei Huang
Anton van den Hengel
Kun Zhang
Javen Qinfeng Shi
Venue: ICLR 2024 poster (2024)
Keywords: Latent Causal Models
Causal Representations
Identifiability
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Causal representation learning aims to unveil latent high-level causal representations from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as \textit{identifiability}. A recent breakthrough explores identifiability by leveraging the change of causal influences among latent causal variables across multiple environments \citep{liu2022identifying}. However, this progress rests on the assumption that the causal relationships among latent causal variables adhere strictly to linear Gaussian models. In this paper, we extend the scope of latent causal models to involve nonlinear causal relationships, represented by polynomial models, and general noise distributions conforming to the exponential family. Additionally, we investigate the necessity of imposing changes on all causal parameters and present partial identifiability results when part of them remains unchanged. Further, we propose a novel empirical estimation method, grounded in our theoretical finding, that enables learning consistent latent causal representations. Our experimental results, obtained from both synthetic and real-world data, validate our theoretical contributions concerning identifiability and consistency.

Published as a conference paper at ICLR 2024
IDENTIFIABLE LATENT POLYNOMIAL CAUSAL MOD- ELS THROUGH THE LENS OF CHANGE
Yuhang Liu', Zhen Zhang', Dong Gong", Mingming Gong 36 Biwei Huang
Anton van den Hengel', Kun Zhang Javen Qinfeng Shi!
Australian Institute for Machine Learning, The University of Adelaide, Australia
School of Computer Science and Engineering, The University of New South Wales, Australia
3 School of Mathematics and Statistics, The University of Melbourne, Australia
4 Halicioğlu Data Science Institute (HDSI), University of California San Diego, USA
5 Department of Philosophy, Carnegie Mellon University, USA
Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates yuhang 1 1iu01 @ Badele a iide.... edu. au
ABSTRACT
Causal representation learning aims unveil latent high-level causal representa- tions from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability. A recent breakthrough explores identifiability by leveraging the change of causal in- fluences among latent causal variables across multiple environments (Liuu al.) 2022). However, this progress rests the assumption that the causal relation- ships among latent causal variables adhere strictly to linear Gaussian models. this paper, we extend the scope of latent causal models to involve nonlinear causal relationships, represented by polynomial models, and general noise distributions conforming to the exponential family. Additionally, investigate the necessity of imposing changes on all causal parameters and present partial identifiability results when part of them remains unchanged. Further, we propose a novel empir- ical estimation method, grounded in our theoretical finding, that enables learning consistent latent causal representations. Our experimental results, obtained from both synthetic and real-world data, validate our theoretical contributions concern- ing identifiability and consistency.
INTRODUCTION
Causal representation learning, aiming to discover high-level latent causal variables and causal struc- tures among them from unstructured observed data, provides prospective way to compensate for drawbacks traditional machine learning paradigms, c.g., the most fundamental limitations that data, driving and promoting the machine learning methods, needs to be independent and identically distributed (i.i.d)) (Schölkopf) 2015). From the perspective of causal representations, the changes in data distribution, arising from various real-world data collection pipelines (Karahan et al. 2016 Frumkin 2016 Pearl et al. 2016)] Chandrasekaran et al. (2021), can be attributed to the changes of causal influences among causal variables (Schölkopf' et al. (02T). These changes observable across multitude of fields. For instance, these could appear in the analysis of imaging data of cells, where the contexts involve batches of cells exposed to various small-moleeut compounds. In this context, each latent variable represents the concentration level of a group of proteins (Chan-) drasekaran al. 202T). An inherent challenge with small molecules their variability in mecha- nisms of action, which can lead to differences in selectivity (Forbes & Krueger 2019). In addition, the causal influences of a particular medical treatment on a patient outcome may vary depending on the patient profiles (Pearl et al. 2016). Moreover, causal influences from pollution to health out- comes, such as respiratory illnesses, can vary across different rural environments (Frumkin) 2016).
Despite the above desirable advantages, the fundamental theories underpinning causal representation learning, the issue of identifiability (i.e., uniqueness) of causal representations, remain a significant challenge. One key factor leading to non-identifiability results is that the causal influences among

---

Published as a conference paper at ICLR 2024
latent space could be assimilated by the causal influences from latent space to observed space, result- ing in multiple feasible solutions (Liu et al., 2022; Adams et al. 202T). To illustrate this, consider two latent causal variables case, and suppose that ground truth is depicted in Figure 1 (a). The causal influence from the latent causal variable 21 to 22 in Figure/ 1 (a) could be assimilated by the causal influence from z to x, resulting in the non-identifiability result, depicted in Figure 1 (b). Efforts to address the transitivity to achieve the identifiability for causal representation learning primarily fall into two categories: 1) enforcing special graph structures (Silva al. 2006 Cai et al. 2019; Xie
et al. 2020 2022 Adams et al.) 2021; Lachapelle et al. 2021), and 2) utilizing the change of causal
influences among latent causal variables (Liu et al. 2022] Brehmer et al. [022]] AAuuaa et al. [023]
Seigal et al. 2022 Buchholz et al. 2023 Varici et al. 2023). The first approach usually requires
special graph structures, i.e., there are two pure child nodes at least for each latent causal variable, as depicted in Figure (c). These pure child nodes essentially prevent the transitivity problem, by the fact that there is an alternative solution to generate the same observational data, the pure child would not be "pure' anymore. For example, if the edge from 21 to 22 in Figure 1 (c) is replaced by two new edges (one from 21 to T2, the other from 21 to Tz)) 22 and 2i are not 'pure' child of 22 anymore. For more details please refer to recent works in Xie al.) (220 2022); [Huang et al. (2022). However, many causal graphs in reality may be more or less arbitrary. beyond the special graph structures. The second research approach permits any graph structures by utilizing the change of causal influences, as demonstrated in Figure 1 (d). To characterize the change, a surrogate vari- able u is introduced into the causal system. Essentially, the success of this approach lies in that the change of causal influences in latent space can not be 'absorbed' by the unchanged mapping from latent space to observed space across (Liu et al. 2022), effectively preventing the transitivity problem. Some methods within this research line require paired interventional data (Brehmer et al.) 2022]. which may be restricted in some applications such as biology (Stark et al., [022)]. Some works require hard interventions or more restricted single-node hard interventions (Ahuja et al.) 2023 Seigal et al. 2022 Buchholz et al. 2023 Varici et al. 2023), which could only model specific types of changes. By contrast, the work presented in Liu et al. (2022) studies unpaired data, and employs soft interventions to model a broader range of possible changes, which could be easier to achieve for latent variables than hard interventions.
The work in Liu et al. 2022) compresses the solution space of latent causal variables up to identifi- able solutions, particularly from the perspective of observed data. This process leverages nonlinear identifiability results from nonlinear ICA (Hyvarinen et al. 2019 Khemakhem et al. 20200; Sorren- son et al. 2020). Most importantly, it relies on some strong assumptions, including causal rela- tions among latent causal variables to be linear Gaussian models, and 2) requiring + (f(E + 1))/2 environments where l is the number of latent causal variables. By contrast, this work is driven by the realization that we can narrow the solution space of latent causal variables from the perspective of latent noise variables, with the identifiability results from nonlinear ICA. This perspective enables us to more effectively utilize model assumptions among latent causal variables, leading to two sig- nificant generalizations: 1) Causal relations among latent causal variables can be generalized to be polynomial models with exponential family noise, and 2) The requisite number of environments can be relaxed to 2 +1 1 environments, a much more practical number. These two advancements narrow the gap between fundamental theory and practice. Besides, of requiring all coefficients within polynomial models to change. We show complete identifiability results if all coefficients change across environments, and partial identifiability results if only part of the coefficients change. The partial identifiability result implies that the whole latent space can be theoretically divided into two subspaces, one relates invariant latent variables, while the other involves variant variables. This may be potentially valuable for applications that focus on learning invariant latent variables to adapt to varying environments, such as domain adaptation or generaliza- tion. To verify our findings, we design a novel method to learn polynomial causal representations in the contexts of Gaussian and non-Gaussian noises. Experiments verify our identifiability results and the efficacy of the proposed approach on synthetic data, image data Ke et al. (2021). and fMRI data.
deeply investigate the assumption
2 RELATED WORK
Due to the challenges of identifiability in causal representation learning, early works focus on learn- ing causal representations in a supervised setting where prior knowledge of the structure of the causal graph of latent variables may be required (Kocaoglu et al. 2018), or additional labels are required to supervise the learning of latent variables (Yang et al. 2021). However, obtaining prior knowledge of
2

---

Published as a conference paper at ICLR 2024
41
a
20
(a) Ground Truth
(b) Possible Solution
(c) Special Structure
(d) Change
Figure I: Assume that ground truth is depicted in Figure (a). Duc the transitivity, the graph structure in Figure 1 (b) is an alternativ solution for (a), leading the non-identifiabiiity result. Figure (c) depicts a special structure where two 'pure' child nodes appear. Figure (d) demonstrates change causal influences, characterized by the introduced surrogate variable u.
the structure of the latent causal graph is non-trivial in practice, and manual labeling can be costly and error-pronee Some works consider temporal constraint that the effect cannot precede the cause has been used repeatedly in latent causal representation learning (Yao et al. 202T Lippe al. [022]] Yao et 2022), while this work aims to learn instantaneous causal relations among latent variables. Besides, there two primary approaches to address the transitivity problem, including imposing special graph structures and using the change of causal influences.
Special graph structures Special graphical structure constraints have been introduced in recent progress identifiability (Silva et al., 20066 Shimizu et al., 2009, Anandkumar et al. 2013 Frot
et al. 2019: Cai et al.] | et al. 2020 2022 2022| Lachapelle et al. [022]) One of representative
graph structures is that there are pure children for each latent causal variables (Xie et al. 2020
2022 Huang et al. 2022). These special graph structures are highly related to sparsity, which implies that a sparser model that fits the observation preferred (Adams et al.) 2021). However, many latent causal graphs in reality may be more less arbitrary, beyond purely sparse graph structure. Differing from special graph structures, this work does not restrict graph structures among latent causal variables, by exploring the change of causal influence among latent causal variables.
The change causal influence Very recently, there have been some works exploring the change of causal influence (Von Kügelgen et al.) 202 I Liu et al. 2022 Brehmer et al.) 2022 Ahuja al.) 2023 Seigal et al. 2022; Buchholz et al. 2023 Varici al. 2023). Roughly speaking, these changes of causal influences could be categorized as hard interventions or soft interventions. Most of them consider hard intervention or more restricted single-node hard interventions (Ahuja et al. 2023 Scigal al. 2022; Buchholz et al., 2023 Varici et al. 2023), , which can only capture some special changes of causal influences. In contrast, soft interventions could model more possible types of change (Liu et al., 2022 Voon Kügelgen
al. 2021). which could be easier achieve in latent space. Differing from the work in Von Kügelgen et al. (2021) that identifies two coarse-grained latent subspaces, e.g., style and content, the work in Liu et al. (2022) aims identify fine-grained latent variables. In this work, we generalize the identifiability results in Liu et al. (2022), and relax the requirement of the number of environments. Moreover, we discuss the necessity requiring all causal influences to change, and partial identifiability results when part of causal influences changes.
IDENTIFIABLE CAUSAL REPRESENTATIONS WITH VARYING POLYNOMIAL CAUSAL MODELS
In this section, we show that by leveraging changes, latent causal representations are identifiable (in- cluding both latent causal variables and the causal model), for general nonlinear models and noise distributions are sampled from two-parameter exponential family members. Specifically, we start by introducing our defined varying latent polynomial causal models in Section 3.1 aiming to facilitate comprehension of the problem setting and highlight our contributions. Following this, in Section 3.2 we present our identifiability results under the varying latent polynomial causal model. This constitutes a substantial extension beyond previous findings within the domain of varying linear Gaussian models (Liu et al. 2022). Furthermore, we delve into a thorough discussion about the necessity of requiring changes in all causal influences among the latent causal variables. We, addi- tionally, show partial identifiability results in cases where only a subset causal influences changes in Section 3.3 further solidifying identifiability findings.
3

---

Published as a conference paper at ICLR 2024
3.1 VARYING LATENT POLYNOMIAL CAUSAL MODELS
We explore causal generative models where the observed data x is generated by the latent causal variables z E IR., allowing for any potential graph structures among z. In addition, there exist latent noise variables n E IR known as exogenous variables in causal systems, corresponding to latent causal variables. We introduce a surrogate variable u characterizing the changes in the distribution of n, as well as the causal influences among latent causal variables Here u could be environment, domain, or time index. More specifically, we parameterize the causal generative models by assuming follows an exponential family given u, and assuming and are generated follows:
(TT.)(nuul
II (u) expl Σ (Tis(m))).,u)))
2: = (pa, u) + n2:
(2)
= f(z) t
with
gi (z, u) = X (u) 282 .... 28.,..)),
(4)
where
in Eq. [ (u) denotes the normalizing constant, and (n:) denotes the sufficient statis- tie for ny whose the natural parameter %h. (h,(() depends on u. Here we focus on two- parameter (e.g.. f1,2}) exponential family members, which include not only Gaussian, but also inverse Gaussian, Gamma, inverse Gamma, and beta distributions.
In Eq. 2 pa, denotes the set of parents of 25-
* In Eq. denotes a nonlinear mapping, and e is independent noise with probability density function (E).
In Eq. where (u) = (u), (2. (u), & represents the Kronecker product with
all distinct entries, e.g., for 2-dimension case, z1 6222 = [=] 22 #122]. Here Au adheres common Directed Acyclic Graphs (DAG) constraints.
The models defined above represent polynomial models and two-parameter exponential family dis- tributions, which include not only Gaussian, but also inverse Gaussian, Gamma, inverse Gamma, and beta distributions. Clearly, the linear Gaussian models proposed in Liu al. (2022) can be seen as special case in this broader framework. The proposed latent causal models, as defined in Eqs. have the capacity to capture a wide range of change of causal influences among latent causal varialle,, including diverse set of nonlinear functions and two- parameeetter exponential fam- ily noises. This expansion in scope serves to significantly bridge the divide between foundational theory and practical applications.
3.2 COMPLETE IDENTIFYABILITY RESULT
The crux of our identifiability analysis lies in leveraging the changes in causal influences among latent causal variables, orchestrated by Unlike many prior studies that constrain the changes within the specific context hard interventions (Brehmer et al. [022] [Ahuja et al.) 2023] Seigal al., 2022 Buchholz al. 2023] Varici al. 2023), approach welcomes and encourages changes. Indeed, our approach allows a wider range of potential changes which can be interpreted soft interventions (via the causal generative model defined in Eqs. 4.
Theorem 3.1 Suppose latent causal variables z and the observed variable x f.ollow the causal gen- erative models defined in Eq. -- Eq. Assume the following holds;
(i) The set {x E X (x)) 0} has measure zero where &e is the characteristic function of the density Pe.
(i)) The function f in Eq. 3 is bijective,
4

---

Published as a conference paper at ICLR 2024
(iii) There exist 2 + values uf i.e., Up u1,... Ugg. such that the matrix
( m u1) - ( uo)..., n = uze) - = uu))
(5)
of size 28 x 2 is invertible. Here nu)) [u),(u)]]e..
(iv) The function class of As.j can be expressed by a Taylor series: for each As.j, de, - 0,
then the true latent causal variables a are related to the estimated latent causal variables &, which learned by matching the true marginal data distribution p(x u), by the following relationship: z P2 + C, where P denotes the permutation matrix with scaling, с denotes a constant vector.
Proof sketch First, we demonstrate that given the DAG (Directed Acyclic Graphs) constraint and the assumption of additive noise in latent causal models in Eq. 4 the identifiability result in Sorrenson et al. (2020) holds. Specifically, it allows us to identify the latent noise variables n up to scaling and permutation, e.g., n Pá + с where denotes the recovered latent noise variables obtained by matching the true marginal data distribution. Building upon this result, we then lever- age polynomial property that the composition of polynomials a polynomial, and additive noise assumption defined in Eq. to show that the latent causal variables can also be identified up to polynomial transformation, I.c., 2 = ( + c where Poly denotes a polynomial function. Fi- nally, using the change of causal influences among z, the polynomial transformation reduced to permutation and scaling, i.e., 2 = På + c. Detailed proof can be found in A.2
be further
Our model assumptions among latent causal variables is typical additive noise model as in Eq. 4 Given this, the identifiability of latent causal variables implies that the causal graph is also identified. This arises from the fact that additive noise models identifiable (Hoyer et al. 2008 PPteers al., 2014), regardless of the scaling on z. In addition, the proposed identifiability result in Theorem 3. represents more generalized form of the previous result in Liu et al. (2022). When the polyno- mial model's degree is set to I and the noise is sampled from Gaussian distribution, the proposed identifiability result in Theorem 3.1 converges to the earlier finding in Liu et al. (2022). Notably, the proposed identifiability result requires only 2 + environments, while the result in Liu et al. (2022) needs the number of environments depending on the graph structure among latent causal variables. In the worst case, e.g., full-connected causal graph over latent causal variables, i.e., ++ (E(E + 1))/2.
3.3 COMPLETE AND PARTIAL CHANGE OF CAUSAL INFLUENCES
The aforementioned theoretical result necessitates that all coefficients undergo changes across vari- ous environments, as defined in Eq. 4 However, in practical applications, the assumption may not hold true. Consequently, two fundamental questions naturally arise: is the assumption necessary for identifiability, in the absence of any supplementary assumptions? Alternatively, can we obtain partial identifiability results if only part of the coefficients changes across environments? In this section, we provide answers to these two questions.
Corollary 3.2 Suppose latent causal variables and the observed variable follow the causal
generative models defined in Eq. [ Eq. { Under the condition that the assumptions (i) ( in Theorem 3.7 are satisfied, if there is an unchanged coefficient in Eq. across environments, is unidentifiable, without additional assumptions.
Proofs The proof of the above corollary can be done by investigating whether we can always construct an alternative solution, different from z, to generate the same observation x, if there an unchanged coefficient across u. The construction can be done by the following: assume that there an unchanged coefficient polynomial for 2,, we can always construct an alternative solution x' by removing the term involving the unchanged coefficient in polynomial gi, while keeping the other unchanged, i.c., 2//- = zj for all * i. Details can be found in A.3
Insights 1) This corollary implies the necessity of requiring all coefficients to change to obtain the complete identifyability result, without introducing additional assumptions. We acknowledge the possibility of mitigating this requirement by imposing specifie graph structures, which is beyond
5

---

Published as a conference paper at ICLR 2024
the scope of this work. However, it is interesting explore the connection between the change of causal influences and special graph structures for the identifiability causal representations in the future. 2) In addition, this necessity may depend on specific model assumptions. For instance, if we use MLPs to model the causal relations of latent causal variables, it may be not necessary to require all weights in the MLPs to change.
Requiring all coefficients to change might be challenging in real applications. In fact, when part of the coefficients change, we can still provide partial identifiability results, as outlined below:
Corollary 3.3 Suppose latent causal variables z and the observed variable x follow the causal generative models defined in Eq. [ Eq. 4 Under the condition that the assumptions Theorem 3.7 are satisfied. for each 24.
in
(a) if it is root node or all coefficients in the corresponding polynomial 9r change in Eq. then the true z, is related to the recovered one 2j. obtained by matching the true marginal data distribution p(x\u), by the following relationship: 2 sžj c, where denotes scaling, e denotes constant,
(b) if there exists an unchanged coefficient in polynomial in Eq. then 2; is unidentifiable
Proof sketch This can be proved by the fact that regardless of the change of the coefficients, two results hold, i.e., z = Poly(2) + c, and n = Pn - C. Then using the change of all coefficients in the corresponding polynomial gi, we can prove (a) For [b) similar to the proof of corollary 3.2
we can construct possible solution 2/ for 2, by removing the term corresponding to the unchanged coefficient, resulting in unidentifiable result.
Insights 1) The aforementioned partial identifiability result implies that the entire latent space can theoretically be partitioned into two distinct subspaces: one subspace pertains invariant la- tent variables, while the other encompasses variant variables. This may be potentially valuable for applications that focus on learning invariant latent variables adapt to varying environments,
such as domain adaptation (or generalization) (Liu et al. 2024). 2) In cases where there exists an unchanged coefficient the corresponding polynomial Ii. although 2; is not entirely identifi- able, we may still ascertain a portion of 2,. To illustrate this point, for simplicity, assume that 22 321 + 2 + 722- Our result (b) shows that 22 is unidentifiable due to the constant coefficient on the right side of the equation. However, the component 21.2(u) + n2 may still be identifiable. While we refrain from presenting a formal proof for this insight in this context, we can provide some elucidation. If we consider 22 as composite of two variables, Za 32, and 26 21,2 ( n2, according to our finding (a), 26 may be identified.
4 LEARNING POLYNOMIAL CAUSAL REPRESENTATIONS
In this section, we translate our theoretical findings into a novel algorithm. Following the work in Liu al. (2022). due permutation indeterminacy in latent space, we can naturally enforee causal order 21 > 22 > ..... > 2z to impose each variable to learn the corresponding latent variables in the correct causal order. As result, for the Gaussian noise, in which the conditional distribu- tions pp,)) where pa, denote the parent nodes of 2,, can be expressed as an analytic form, we formulate the prior distribution as conditional Gaussian distributions. Differing from the Gaussian noise, non-Gaussian noise does not have an analytically tractable solution, in general. Given this, we model the prior distribution of p(z)uz ()) by p(X, n/u). As a result, we arrive at:
f
p(21 u)) II p(z Zen u) = II (N(,,, 82, (u)), = (
if n Gaussian
p(z|u)=
(6)
I T p(n:/u),
if n ~ non Gaussaanii
where (ue, (u), 82, (u)) denotes the Gaussian probability density function with mean May (u) and variance 82 (u). Note that non-Gaussian noises typically tend result in high variance gradients. They often require distribution-specifee variance reduction techniques to be practical, which is be- yond the scope of this paper. Instead, we straightforwardly the PyTorch (Paszke et al. 2017)

---

Published as a conference paper at ICLR 2024
implementation of the method of Jankowiak & Obermeyer (2018). which computes implicit repa- rameterization using a closed-form approximation of the probability density function derivative. In our implementation, for inverse Gaussian and inverse Gamma distributions. Therefore, we present the results of Gamma, and beta distributions in experiments.
found that the implicit reparameterization leads to high- high-variance gradients
Prior on coefficients p(Ag,) We enforce two constraints on the coefficients, DAG constraint and sparsity. The DAG constraint is ensure a directed acyelic graph estimation. Current methods usually employ relaxed DAG constraint proposed by /Zheng et al. (2018) to estimate causal graphs, which may result in a cyclic graph estimation due the inappropriate setting of the regularization hyperparameter. Following the work in Liu et al. 22022), we can naturally ensure directed acyclic graph estimation by enforcing the coefficients matrix A(u)² to be a lower triangular matrix cor- responding to fully-connected graph structure, due permutation property in latent space. In addition, to prune the fully connected graph structure to select true parent nodes, we enforce a spar- sity constraint on each (u). In our implementation, we simply impose a Laplace distribution each (u) other distributions may also be flexible, e.g. horseshoe prior (Carvalho et al. 2009) or Gaussian prior with mean and variance sampled from a uniform prior (Liu et al. 2019).
Variational Posterior We employ variational posterior to approximate the true intractable poste- rior of p(z[x, u). The nature of the proposed prior in Eq. gives rise to the posterior:
(q(2)/u x) II a Zec, u. x), a (A,alx, u)) Ii ((s//u, x), if
if Gaussian
qz//u.
Gaussian
where variational posteriors q<z//zz, x)x ((A,,)) and q(n, uu) employ the same distribution as their priors, so that an analytic form of Kullback-Leible divergence between the variational poste- rior and the prior can be provided. As result, we can arrive at simple objective:
max (g(te) ()) (p(x/z, z u)) - (rx(x( u)llp(z u)) - (xI(g u)]lp(A|u)), (8)
where DEL denotes the KL divergence. Implementation details can be found in Appendix A.6
5 EXPERIMENTS
Synthetic Data We first conduct experiments on synthetic data, generated by the following pro- cess: we divide latent noise variables into M segments, where each segment corresponds to one value of u as the segment label. Within each segment, the location and scale parameters are respec- tively sampled from uniform priors. After generating latent noise variables, we randomly generate coefficients for polynomial models, and finally obtain the observed data samples by invertible nonlinear mapping on the polynomial models. More details can be found in Appendix A.5
We compare the proposed method with vanilla VAE (Kingma & Welling 2013). 9- B-E (Higgins ctt al. 2017). identifiable VAE (iVAE) (Khemakhem et al. (020)) Among them, iVAE is able to iden- tity the true independent noise variables up permutation and scaling, with certain assumptions. 3-VAE has been widely used in various disentanglement tasks, motivated by enforcing independence among the recovered variables, but has no theoretical support. Note that both methods assume that the latent variables are independent, and thus they cannot model the relationships among latent vari- ables. All these methods are implemented in three different settings corresponding to linear models with Beta distributions, linear models with Gamma noise, and polynomial models with Gaussian noise, respectively. To make fair comparison, for Gaussian noise, all these methods use the PyTorch (Paszke et al., (017)) implementation of the method of Jankowiak & Obermeyer (2018) to compute implicit reparameterization. We compute the mean of the Pearson correlation coefñ- cient (MPC) to evaluate the performance of our proposed method. Further, we report the structural Hamming distance (SHD) of the recovered latent causal graphs by the proposed method.
Figure 2 shows the performances of different methods on different models, in the setting where all coefficients change across u. According to MPC, the proposed method with different model assump- tions obtains satisfactory performance, which verifies the proposed identifiability results. Further, Figure [ shows the performances of the proposed method when part of coefficients change across u, for which we see that unchanged weight leads to non-identifaabiliyy results, and changing
7

---

Published as a conference paper at ICLR 2024
68-
66-
LinearBeta
LinearGamma
PolyGaussian
SHD
Figure 2: Performances of different methods on linear models with Beta noise, linear models with Gamma noise, and polynomial models with Gaussian noises. In terms of MPC, the proposed method performs better than others, which verifies the proposed identifiablity results. The right subfigure shows the SHD obtained by the proposed method in different model assumptions.
0.41
0.8
0.90
0.7
0.83
0.6-
0.5
0.82
0.75
Lasend Caunal Var abres
Laserd Caunal Vataares
Lateni Caussl Vartacles
21 + 22
2/2 > 23
23
24
Figure 3: Performances of the proposed method with the change of part of weights, on linear models with Beta noise. The ground truth of the causal graph is 21 + 2/ + + From left to right: keeping weight unchanged. Those results are consistent with the analysis of partial
+ 22, &2 + and 23 +
identifiability results in corollary 3.3
weights contribute to the identifiability of the corresponding nodes. These empirical results are consistent with partial identifiability results in corollary 333
Image Data We further verify the proposed identifiability results and method on images from the chemistry dataset proposed in Ke et a (2021), which corresponds chemical reactions where the state of an element can cause changes to another variable's state. The images consist of a number of objects whose positions are kept fixed, while the colors (states) of the objects change according to the causal graph. To meet our assumptions, we use weight- variant linear causal model with Gamma noise to generate latent variables corresponding to the colors. The ground truth of the latent causal graph is that the 'diamond" (e.g., =1) causes the 'triangle' (e.g.. #2), and the 'triangle' causes the 'square' (e.g., zz). A visualization of the observational images can be found Figure Figure 5 shows the MPC obtained by different methods. The proposed method performs better than others. The proposed method also learns the correct causal graph as verified by intervention results in Figure 6 i.e., 1) intervention
21 ('diamond') causes the change of both 22 ('triangle') and 23 ('square') ), 2) intervention on 22 only causes the change of 23, 3) intervention on 23 can not affect both 21 and z2. These results are consistent with the correct causal graph, i.e., 21 22 + 23. Due to limited space, more traversal results on the learned latent variables by the other methods can be found in Appendix A.7 For these methods, since there is no identifiability guarantee, found that traversing on each learned variable leads to the colors of all objects changing.
Figure 4: Samples from the image dataset generated by modifying the chemistry dataset in Ke all 2022 The colors (states) the objects change according the causal graph: the 'diamond" causes the "Iriangle" and the 'triangle" causes the 'square', i.e., 21 -> Z4 23.
8

---

Published as a conference paper at ICLR 2024
Z1
Z2
2:
Z1
Z:
22
22
21
0.963
0.005
0.115
2:
0.067
0.582
0.628
2
0.053
0.742
0.228
2:
0.953
0.035
0.068
22
0.008
0.702
0.491
0.958
0.065
0.046
22
0.858
0.265
0.046
22
0.137
0.852
0.392
2,
0.085
0.421
0.803
0.117
0.429
0.765
2
0.117
0.429
0.765
2.
0.186
0.401
0.871
iVAE
3-VAE
VAE
Ours
Figure 5: MPC obtained by different methods the image dataset, the proposed method performs better than others, supported by our identifiability.
Intervention ž1
Intervention 22
Intervention on 23
Figure 6: Intervention results obtained by the proposed method the image data. From left to right: inter- ventions on the learned 21, #2, 23, respectively. The vertical axis denotes different samples, The horizontal axis denotes enforcing different values on the learned causal representation.
IMRI Following Liu et al. (2022 we further apply the proposed method to fMRI hippocam- pus dataset (Laumann & Poldrack] 2015), which contains signals from six separate brain regions: perirhinal cortex (PRC), parahippocampal cortex (PHC), entorhinal cortex (ERC), subiculum (Sub), CA1, and CA3/Dentate Gyrus (DG). These signals were recorded during resting states over a span of 84 consecutive days from the same individual. Each day's data considered a distinct instance, resulting in 84-dimensional vector represented latent causal variables, we treat the six signals as latent causal variables. transform them into ob- served data, subject them to random nonlinear mapping. Subsequently, we apply our proposed method to the transformed observed data. We then apply the proposed method to the transformed observed data to recover the latent causal variables. Figure 7 shows the results obtained by the proposed method with different model assumptions. We can see that the polynomial models with Gaussian noise perform better than others, and the result obtained by linear models with Gaussian noise is suboptimal. This also may imply that 1) Gaussian distribution is more reasonable to model the noise in this data, linear relations among these signals may be more dominant than nonlinear.
Given primary interest in uncovering
PRS
DG
PRB
DG
PRG
DG
PRS
ERG
GAT
ERG
CAT
ERC
CAT
ERC
CA1
MPC
LinBeta
LinGamma
LinGaussian
PolyGaussiad
Figure 7: MPC obtained by the proposed method with different noise assumptions. Blue edges are feasible given anatomical connectivity, red edges are not, and green edges are reversed.
CONCLUSION
Identifying latent causal representations is known be generally impossible without certain as- sumptions. This work generalizes the previous linear Gaussian models to polynomial models with two-parameter exponential family members, including Gaussian, inverse Gaussian, Gamma, inverse Gamma, and Beta distribution. We further discuss the necessity of requiring all coefficients in poly- nomial models to change to obtain complete identifiability result, and analyze partial identifiability results in the setting where only part of the coefficients change. We then propose a novel method to learn polynomial causal representations with Gaussian or non- non-Gaussian noise. Experimental results on synthetic and real data demonstrate our findings and consistent results. Identifying causal rep- resentations by exploring the change of causal influences is still an open research line. In addition, even with the identifiability guarantees, it is still challenging learn causal graphs in latent space.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
We are very grateful to the anonymous reviewers for their help in improving the paper. YH was partially supported by Centre for Augmented Reasoning. DG was partially supported by an ARC DECRA Fellowship DE230101591. MG was supported by ARC DE2 0E210101624. KZ would like to acknowledge the support from NSF Grant 2229881, the National Institutes of Health (NIH) under Contract 501HL119805, and grants from Apple Inc., KDDI Research Inc., Quris AI, and Infinite Brain Technology.
REFERENCES
Jeffrey Adams, Niels Hansen, and Kun Zhang. Identification of partially observed linear causal models: Graphical conditions for the con-gaussian and heterogeneous cases. In NeurIPS, 2021.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa- tion learning. In International Conference on Machine Learning, pp. 372-407, PMLR, 2023.
Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In ICML, pp. 249-257, 2013.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal represen- tation learning. arXiv preprint arXiv:2203. /6437, 2022.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general non- linear mixing. arXiv preprint (tti::330600233333 2023.
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. In NeurlPS, 2019.
Carlos M Carvalho, Nicholas G Polson, and James G Scott. Handling sparsity via the horseshoe. In Artificial intelligence and statisties, pp. 73-80. PMLR, 2009.
Srinivas Niranj Chandrasekaran, Hugo Ceulemans, Justin D Boyd, and Anne Carpenter. Image- based profiling for drug discovery: due for machine-leariing upgrade? Nature Reviews Drug Discovery, 20(2): 155-159, 2021.
Miriam K Forbes and Robert Krueger. The great recession and mental health the united states. Clinical Psychological Science, 7(5):900-99) 2019.
Benjamin Frot, Preetam Nandy, and Marloes H Maathuis. Robust causal structure learning with some hidden variables. Journal of the Royal Statistical Society: Series B (Statistical Methodol- ogy), 81(3):459-487, 2019.
Howard Frumkin. Environmental health: from global local. John Wiley Sons, 2016.
I. Higgins, Loic Matthey, A. Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In /CLR, 2017.
Patrik O Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, Bernhard Schölkopf, et al. Nonlinear causal discovery with additive noise models. In NeurIPS, volume 21, pp. 689-696. Citeseer, 2008.
Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang, Latent hicrarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 35:5549-5555 01154995566,222 2022.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868, PMLR, 2019.
Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparameterizattion trick.
In International conference machine learning, pp. 2235-2244 PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Samil Karahan, Merve Kiline Yildirum, Kadir Kirtac, Ferhat Sukru Rende, Gultekin Butun, and Hazim Kemal Ekenel. How image degradations affect deep enn-based face recognition? In 2016 international conference of the biometrics special interest group (BIOSIG), pp. 1-5. IEEE, 2016.
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery visual model based reinforcement learning. arXiv preprint arXiv:2107.00848, 2021.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In AISTAS, pp. 2207-2217. PMLR, 2020.
Diederik P Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katic Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. arXiv preprint 200. 10098, 2021.
Timothy O.
Laumann and Russell A. Poldrack, 2015.
URL https:/
Gataset/as000031/
Phillip Lippe, Sara Magliacane. Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learming. pp. 13557-13603. PMLR. 2022.
Yuhang Liu, Wenyong Dong, Lei Zhang, Dong Gong, and Qinfeng Shi. Variational bayesian dropout with a hierarchical prior. In CVPR, 2019.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- variant latent causal models. arXiv preprint arXiv:2208. 14153, 2022.
Yuhang Liu, Zhen Zhang. Dong Gong, Mingming Gong, Biwei Huang, Anton den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifiable latent causal content for domain adaptation under latent covariate shift. arXiv preprint arXiv:2208. 14161, 2024.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS workshop, 2017.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer.
John Wiley & Sons, 2016.
Jonas Peters, Joris M. Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. JMLR, 15(58):2009-2055 2014.
Bernhard Schölkopf. Learning to see and act. Nature, 518(7540):486 487, 2015.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-65) 2021.
Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211. 2022.
Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of linear non-gaussian acyclic models for latent factors. Neurocomputing. 72(7-99:204-20227, 2009.
Ricardo Silva, Richard Scheines, Clark Glymour, Peter Spirtes, and David Maxwell Chickering. Learning the structure of linear latent variable models. JMLR, 7(2), 2006.

---

Published as a conference paper at ICLR 2024
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible- flow networks (gin). arXiv preprint arXiv:2001.04453 04872, 2020.
Stefan G Stark, Joanna Ficek, Francesco Locatello, Ximena Bonilla, Stéphane Chevrier, Franziska Singer, Tumor Profiler Consortium, Gunnar Rätsch, and Kjong- -oong-Van Lehmann. Scim: universal single-cell matching with unpaired feature sets. Bioinformatics, 36, 12 2020.
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXiv:2301.055 2023.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in neural information processing systems, 2021. 2021.
Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. In NeurlPS, 2020.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identifica- tion of linear non-gaussian latent hierarchical structure. In International Conference Machine Learning, pp. 24370-24333 PMIR, 2022.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. In CVPR. 2021.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Learning latent causal dynamics. arXiv:2202.04828, 2022.
arXiv preprint
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. In NeurlPS, 2018.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
THE RESULT IN (IU ET AL.. 2022
For comparison, here we provide the main model assumptions and result in the work by (Liu al. 2022 It considers the following causal generative models:
:~ N (mr, (u), mx2(u)),
== X (u)(z) + +
(10)
x f(z) +
(11)
Theorem A.1 Suppose latent causal variables and the observed variable follow the generative models defined in Eq. 9 Eq. 11 with parameters (f, X, 17). Assume the following holds:
(i) The set xx E Xlve (x) 0} has measure (i.e., has the most countable number of elements), where &e is the characteristic function of the density Pe-
(ū) The function f in Eq. 17 is bijective.
(iii) There exist 28 +1 distinct points Un,0, Un.1, .... un such that the matrix
Ln = (17( (u,,,))) (ul)),,.,777,,22 - (nn,)))
(12)
of size x 28 is invertible.
(iv) There exist l + distinct points Uz,0s Uz,1, Uz such that the matrix
L (7z(1 7z Me(uu, (u2.u)..... M2 (u(u '7 (U2,0)
(13)
of size k is invertible.
(v) The function class of X4.5 can be expressed by Taylor series: for each >4.j. )s,
then the recovered latent causal variables 2, which are learned by matching the true marginal data distribution p(r)u), are related to the true latent causal variables z by the following relationship: P2 c, where P denotes the permutation matrix with scaling, e denotes constant vector.
Here 7) denote sufficient statistic of distribution of latent noise variables n, The denote sufficient statistic of distribution of latent causal variables z. h denotes the number of the sufficient statistic of Please refer to (Liu al. 2022) for more details.
Compared with the work in (Liu et al. 2022), this work generalizes linear Gaussian models in Eq. 10 polynomial models with two-pan poo-paramtter exponential family, defined in Eq. 2 In addition, this work removes the assumption (iv), which requires the number of environments highly depending on the graph structure. Moreover, both the work in (Liu et al. 2022) and this work explores the change of causal influences, in this work, we provide analysis for the necessity of requiring all causal influence change, and also partial identifiability results when part of causal
influences changes. This analysis enables the research line, allowing causal influences to change, more solid.
13

---

Published as a conference paper at ICLR 2024
A.2 THE PROOF OF THEOREM 3.1
For convenience, we first introduce the following lemmas.
Lemma A.2 2 can be expressed as a polynomial function with respect to n, i.e, 2 = h(n, u), where h denote polynomial, and h is also a polynomial functiin..
Proof can be easily shown by the following: since have established that 2; depends on its parents and n, as defined in Eqs. 2 and 4 we can recursively express 2, in terms of latent noise variables
relating its parents and n, using the equations provided in Eqs. 2 and Specifically, without loss of the generality, suppose that the correct causal order is 21 > 22
26, we have:
n1
hn (m))
22
g2(21) +12 g2(n1, u)(ni,u)a0 n22
h2(m ,722 23 ga(21, (2 u) 12; u) + T&3
(14)
where h(n, u) = (n1, u), (121, n2, u), (11, n2, Ry, u).... By the fact that the composition of polynomials still polynomial, and repeating the above process for each 2; can show that can be expressed as polynomial function with respect to i.e., additive noise models and DAG constraints, can be shown that the Jacobi determinant 1, and thus the mapping h is invertible. Moreover, h -1 can be recursively expressed in terms of z according to Eq. as follows:
h(n, u). Further, according to the
equals
mh
21
h (m1)
n2 = 22 (2 u) = 22 92(21,
23 95(21,02(M1, u) + n2, u) 23
u) + (22 92(21, ())), u) Ag 05'((1182.0 (21.22.23.4))
(15)
Again, since the composition of polynomials is still a polynomial, the mapping h is also a poly- nomial.
Lemma A.3 The mapping from to x, e.g., f h, is invertible, and the Jacobi determinant det Jrchl = Jel det Jul = [det.r]. and thus det J | J Fohl = det J which do not depend on u.
Proof can be easily shown by the following: Lemma A.2 has shown that the mapping h, from n to z, is invertible. Together with the assumption that f is invertible, the mapping from n to x is invertible. In addition, due the additive noise models and DAG constraint as defined Eq. 14 we can obtain
| det J, 1.
Lemma A.4 Given the assumption (iv) Theorem 3.1 the partial derivative of l (,(nn, Riy u) in
Eq. 14
with respect ne, where i, equals when u i.e., 0h, (fi..... u= 0.
Orgg
Since the partial derivative of the polynomial ha(n1..., n4, is still polynomial whose coeffi- cients are scaled by X₂ (u), as defined in Eq. 14 and by using the assumption (iv) we can obtain the result.
The proof of Theorem 3.1 is done in three steps. Step I is to show that the identifiability result in (Sorrenson al. 2020 holds in setting, i.e., the latent noise variables be identified up
14

---

Published as a conference paper at ICLR 2024
to component-wist wise scaling and permutation, Pn + Using this result, Step II shows that can be identified up to polynomial transformation, i.c., z = Poly(2) + c. Step III shows that the polynomial transformation in Step II can be reduced to permutation and scaling, using Lemma A.4
P2 c, by
Step I: Suppose we have two sets of parameters 0 = (,, 17) and 0 = (f. f. A. n) correspond-
ing the same conditional probabilities, i.e., (..) P( (2.)) (x(u) for all pairs (x, u), where ' denote the sufficient statistic of latent noise variables n. Due the assumption assumption [it] and the fact that h is invertible (e.g., Lemma A.2). by expanding the conditional
the
probabilities (More details can be found in Step for proof of Theorem I in (Khemakhem et al. ). we have:
log det J ((ro))- + m) (n|u) = log det (fah)) 1xx) + logP(T..) (ń/u),
(16)
Using the exponential family as defined in Eq. T we have:
log det J [xx] T ((f - h) (x))) -lg II 2,(u)
(17)
log det J (foh)- (x)) T ((f o (xx)) log II 2,(u),
(18)
By using Lemma A.3 Eqs. 17 18
can be reduced
log Jp-1 (x) T o(foh) (x)))) log II [u)
i
log| det [x-- T ((f (x)) - log II (u).
(19)
Then by expanding the above at points uy and ug, then using Eq. 19 at point u, subtract Eq. 19 at point Un we find:
(()). (u)) + Σ log Z,(w)) (f()), (u))
Zu(u))
log
(20)
(u:
Here 7u,) n7;)) () By assumption (iii) and combining the 28 expressions into a single matrix equation, we can write this in terms of L from assumption (iii
L''T(n) = L' + b.
(21)
Since L invertible, we can multiply this expression by its inverse from the left to get:
T((f o ((ffoh) hx)) = (T(( ( (x)) +
(22)
Where A = (L")-'ET According to lemma in (Keemahhem et al. 2020) that there exist k distinct values n to n such that the derivative (' "...,'((m,) are linearly independent, and the fact that each component of T is univariate, we can show that A is invertible.
Since we assume the noise be two-parameter exponential family members, Eq. 22 can be re- expressed as:
T (n) (2(n)
) =
T (n) (n) ) c,
(23)
Then, we re-express T2 in term of TT, e.g. (2(n)) ((Ti(::)) where is a nonlinear mapping. As a result, we have from Eq. 23 that: (a) Ti (m:)) can be linear combination of (n) and (2),, and (b) E(T, (n,)) can also be linear combination of (n) and (n). This implies the contradiction that both Ti (n,) and its nonlinear transformation ((Ti (m+)) can be expressed by linear combination of Í, (ń) and Ty (n). This contradiction leads to that A can be reduced to permutation matrix P (See APPENDIX C in Sorrenson et al. 2020) for more details):
Pn +
(24)
15

---

Published as a conference paper at ICLR 2024
where P denote the permutation matrix with scaling, denote constant vector. Note that this result holds for not only Gaussian, but also inverse Gaussian, Beta, Gamma, and Inverse Gamma (See Table in (Sorrenson et al. 2020)).
Step 11:By Lemma A.2
we can denote and by:
h(n),
(25)
h(a),
(26)
where defined in A.2 Replacing n and in Eq. 24 by Eq. have:
25 and Eq.
26
respectively, we
h = Ph
2 +
(27)
where h (as well as h) are invertible supported by Lemma A.2
We can rewrite Eq. 27 as:
hPPh (2) + c).
(28)
Again, by the fact that the composition of polynomials is still polynomial, we can show:
Poly(2) +
(29)
Step III Next, Replacing and 2 in Eq. 29 by Eqs. 24 and 26
h h(Pn + Poly(h(fi) + +
(30)
By differentiating Eq. with respect to
JnP = J.
(31)
Without loss of generality, let us consider the correct causal order 21 > 22 > ...,>> 2ς so that Jn and Jn are lower triangular matrices whose the diagonal are 1, and P diagonal matrix with elements $1,1; $2,2; 83,3,
Elements above the diagonal of matrix Joy Since Jn are lower triangular matrices, and P is diagonal matrix, Joly must be a lower triangular matrix.
Then by expanding the left side of Eq. 31
have:
81.1
8he (02 m22.0)
J.P
$1,1
Bag
$2,2
(32)
Bhg (m) .ng mg.
(mg..nnngg..... Onz
$1.1
Ona
$2,2
$3,3
by expanding the right side of Eq.
31 we have:
Jpoly1.1
0
Juutyz +
J1ou2222
(h2(n1,m2..)
Jrolya.2
0
J JpnyJ f
Oms
J
+ Ed=
Jrays
ah. (..)))
J
J Jolys.
Bha (o..n.
J
8m1
Ong
(33)
The diagonal of matrix Jraly By comparison between Eq.
32 and Eq.
33
we have J 8
Elements below the diagonal of matrix J Lemma A.4 for all > j have Jroly 0.
By comparison between Eq. and Eq.
33 and
As result, the matrix Jol in Eq. 31 equals to the permutation matrix P. which implies that the polynomial transformation Eq. reduces to a permutation transformation.
= Pa + c'.
(34)
16

---

Published as a conference paper at ICLR 2024
A.3 THE PROOF OF COROLLARY 3.2
To prove the corollary, we demonstrate that it is always possible to construct an alternative solution, which is different from the z, but capable of generating the same observations x, if there is an unchanged coefficient across u. Again, without loss of the generality, suppose that the correct causal order is 21 22 > .... > Zg. Suppose that for 2;, there an unchanged coefficient Ay. related to the term of polynomial 2 across u, where & denotes polynomial features created by raising the variables related parent node to an exponent. Note that since we assume the correct causal order, the term @ only includes 2j where j < i. Then, we can always construct new latent variables z as: 2k. and zj = 24 Aj,e Given this, we can construct a polynomial mapping
for all + i, z' M, so that
M(z') Z,
(35)
where
M(2))
(36)
2/ + Xy 2/11 2++1
where for all + z'k 24 the right. clear the Jacobi determinant of the mapping M always equals 1, thus the mapping M is invertible. In addition, all the coefficients of the polynomial mapping M are constant and thus do not depend on u. As result, we can construct a mapping f M as the mapping from z' to x, which is invertible and do not depend u, and can create the same data x generated by f(z). Therefore, the alternative solution z' can lead to a non-identifiabilit result.
A.4 THE PROOF COROLLARY 3.3
Since the proof process in Steps and II in A.2 do not depend on the assumption of change of causal
influence, the results in both Eq. 32 and Eq. 33 hold. Then consider the following two cases.
For the case where is root node all coefficients all paths from parent nodes 2;
change across by using Lemma A.4 i.e., 0h;(m) ................................
0 and (f.(nq
On
On,
0 for all i' and by comparison between Eq. 32 and Eq. 33 we have: for all > j we have Jrny.. = 0, which implies that we can obtain that 2, A,, c.
If there exists unchanged coefficient in all paths from parent nodes 2; across u1 then by the proof of corollary 3.2] i.e., 2/ can be constructed as a new possible solution to replace z; by removing the unchanged coefficient A9,5, resulting in the non-identifiability result. This is also can be proved in another way, e.g., comparison between Eq, 32 and Eq. 33 Suppose that the coefficient is A3.a related to the parent node with the index Given that,
we have: (M.(A1... include constant term A3.5- Again, by using the Lemma A.4 dni, and by comparison between Eq. 32 and Eq. 33 we can only arrive that Sk, JPoby.
As result, 2 will be expressed as combinatron of 2 and ž.
17

---

Published as a conference paper at ICLR 2024
A.5 SYNTHETIC DATA
Data For experimental results on synthetic data, the number of segments is 30, and for each seg- ment, the sample size is 1000, while the number (e.g,, dimension) of latent causal noise variables 2,3,4,5 respectively. Specifically, for latent linear causal models, we consider the following struc- tural causal model:
B(a, 5),
if Beta if Gamma
Th g(a,
(37)
21 = 71
(38)
22 21,2(u)21 72
(39)
23 22 + 133
(40)
24 ( 144
(41)
25 = A (u)23 + 73.
(42)
(43)
where both and 33 for both Beta and Gamma distributions, are sampled from uniform distribution
00.12.0), and (u) are sampled from a uniform distribution [-1.0, -0.5] [0.5, 1.0]. For latent polynomial causal models with Gaussian noise, we consider the following structural causal model:
74 1~ N(a, B),
(44)
31 01
(45)
22 = \1.2 2 122
(46)
X2.3 (2.3(1)22 +
(47)
24 = A3,4 22 14
(48)
25 = As,5 (u) + ng.
(49)
(50)
where both and B for Gaussian noise are sampled from uniform distributions [-2.0, 2.0] and
[0.1,2.0], respectively. (u) are sampled from a uniform distribution [-1.0, -0.5] [0.5, 1.0).
A.6 IMPLEMENTATION FRAMEWORK
Figure 8 depicts the proposed method learn polynomial causal representations with non- nan-Gaussian noise. Figure 9 depicts the proposed method to learn polynomial causal representations with Gaus- sian noise. For non-Gaussian noise, since there is generally no analytic form for the joint distribution of latent causal variables, we here assume that p(z[u) p( p(A(u))p(nju) as defined in Eq, 6 Note that this assumption may be true in general, since it destroys independent causal mechanisms generating effects in causal systems. For experiments on the synthetic data and fMRI data, the en- coder, decoder, MLP for X, and MLP for prior are implemented by using 3-layer fully connected networks and Leaky-Rel.U activation functions. For optimization, we use Adam optimizer with learning rate 0.001. For experiments on the image data, we also use 3-layer fully connected network and Leaky-Re ReLU activation functions for À and the prior model. The encoder and decoder can be found in Table I and Table 2 respectively.
18

---

Published as a conference paper at ICLR 2024
Input
Ceaky-Ree.U(CCon22d..3 32, stride=2, padding=11 I))
Leaky-ReLU(Coon22(3322 32, 4, stride=2, padding=1)
Leaky--Re_U/(Conv2dd(32. 32, 4, stride=2, padding=1))
Leaky-Rel.U((Con22((222 32, 4, stride=2, padding=1)
Leaky-RelU/Limenn322333222 size(u), 30))
Leaky- Ceakk-ReLU(Linear(31) 30))
Linear(30. 3*2)
Table 1: Encoder for the image data.
Latent Variables
Leaky-Rel.U(L.mmarrg 30))
Leaky-ReLU/Limear(300 30))
Lealy-ReLUULLincar(300, 32 x 32 x4))
PekkyyKeeLUUCConTTTTssss2222222 32, 4, stride=2, paddin=1)))
Leaky- Raaly-ReLLUCConTTTTappsssssss 32, 4, stride=2, padding=1))
Soaly--eeLL(CCnn//Taanpppsssss 32, 4, stride=2, padding=1))
Conv'l hwmmTraaspooe223222..2 3, stride=2, padding=1)
Table 2: Decoder for the image data.
MLP
Location and Perameters
((n)
KL(g(n| u) (ppx(/)
Location Parameters
MLP
p (((())
KL u) ui)
MLP
((((w) X. U)
2((u)
Post Payle,s)+nn
Decoder
Location Parameters
Encoder
q X, u)
Figure 8: Implementation Framework to learn polynomial causal representations with non-Gaussian noise.
MLP
Location Peremeters
p(n)
xL(g(z|x,u) lplzz
Location and Parameters p (((())
MLP
x) m)
Location and Parameters
q(A(w) u)
Au
Sy Polyt o Polyiai)) +ny
Decoder
Location and Parameters q X
Encoder
Figure 9: Implementation Framework to learn polynomial causal representations with Gaussian noise, for which analytic form for the prior and posterior of latent causal variables can be provided.
19

---

Published as a conference paper at ICLR 2024
Traversals on 21
Traversals on 22
Traversals on 23
Traversal results obtained by 3-VAE the image data. The vertical axis denotes different samples, The horizontal axis denotes enforcing different values on the learned causal rep- resentation. The ground truth of the latent causal graph is that the 'diamond' (e.g., 21) causes the 'triangle' (e.g.. #2), and the 'triangle' causes the 'square' (e.g.. z3). We can see that the change of each learned variable results in the change of color of all objects.
Figure 10:
Traversals on ø
Traversals on 22
Traversals on 23
Figure 11: Traversal results obtained by VAE on the image data. The vertical axis denotes different
samples, The horizontal axis denotes enforcing different values on the learned causal representation. The ground truth of the latent causal graph is that the 'diamond' (e.g., 21) causes the 'triangle' (e.g.. 22), and the 'triangle' causes the 'square' (e.g., 23). We can see that the change each learned variable results in the change of color of all objects.
A.7 TRAVERSALS ON THE LEARNED VARIABLES BY IVAE, 3- VAE, AND VAE
Since there is no theoretical support for both 3-VAB and VAE, these two methods can not disentangle latent causal variables. This can be demonstrated by Figure I0 and Figure [T which shows that traversal on each learned variable leads to the change of colors of all objects. It is interesting to note that iVAE has the theoretical guarantee of learning latent noise variables. And since we assume additive noise models, e.g., 21 711, iVAE is able identify 21. This be verified from the result obtained by iVAE shown in Figure 5 which shows that the MPC between recovered z and the true one is 0.963. However, iVAE can identify z2 and 23. since these are causal relations among the latent causal variables. We can see from Figure [2] the changes of by and 23 lead to the change of colors of all objects.
A.8 FURTHER DISCUSSION ON THE PARTIAL IDENTIHABILITY COROLLARY
While demanding a change in all coefficients may pose challenges in practical applications, Corol- lary 3.3 introduces partial identifiability results. The entire latent space can theoretically be parti- tioned into two distinct subspaces: an invariant latent subspace and variant latent subspace. This partitioning holds potential value for applications emphasizing the learning of invariant latent vari- ables to adapt to changing environments, such as domain adaptation (or generalization), as discussed
20

---

Published as a conference paper at ICLR 2024
Traversals on 21
Traversals on 22
Traversals on 23
Traversal results obtained by iVAE the image data. The vertical axis denotes different samples, The horizontal axis denotes enforcing different values on the learned causal representation. The ground truth of the latent causal graph is that the 'diamond' (e.g., 21) causes the 'triangle' (e.g., #2), and the 'triangle' causes the 'square' (e.g.. 23). We can see that the change of each learned variable results in the change of color of all objects.
Figure 12:
in the main paper. However, the impact of partial identifiability results the latent causal graph structure remains unclear.
We posit that if there no interactions (edges) between the two latent subspaces in the ground truth graph structure, the latent causal structure in the latent variant space can be recovered. This recovery possible since the values of these variant latent variables can be restored up to component-wise permutation and scaling. In contrast, when no interactions exist between the two latent subspaces in the ground truth graph structure, recovering (part of) the latent causal structure becomes highly chal- lenging. We believe that the unidentifiable variables in the invariant latent subspace may influence the latent causal structure in the variant latent subspace.
We hope this discussion can inspire further research to explore this intriguing problem in the future.
A.9 FURTHER DISCUSSION ON MODEL ASSUMPTIONS IN LATENT SPACE.
In our approach, we selected polynomial models for their approximation capabilities and straightfor- ward expressions, streamlining analysis and facilitating the formulation of sufficient changes. While advantageous, this choice high-order terms in polynomials during optimization. Overall, we think that model assumptions can be extended to a broader scope than polynomials, including non-paran non-parametric models. This extension is contingent on deeming changes in causal influences as sufficient. The crucial question in mov- ing towards more general model assumptions revolves around defining what constitutes sufficient changes in causal influences.
without recognized limitations, notably the challenges introduced by
A. UNDERSTANDING ASSUMPTIONS IN THEOREM
Assumptions ( are motivated by the nonlinear ICA literature (Khemakhem et al. 2020). which is to provide a guarantee that we can recover latent noise variables up permutation and scaling transformation. The main Assumption (ii) essentially requires sufficient changes in latent noise variables to facilitate their identification. Assumption (iv) is derived from the work by Liu et al.
2022) and is introduced to avoid a specific case: X (u) = (u) + b, where is a non-zero constant. To illustrate, if 22 = (u) + b)2, + n2, the term b2, remains unchanged across all u, resulting in non-identifiability according to Corollary 3.3 While assumption (iv) is sufficient for handling this specific case, it may not be necessary. We anticipate the proposal of a sufficient and
necessary condition in the future address the mentioned special case.
21

---

Number of Latent Causal Variables Published as a conference paper at ICLR 2024
1.0
1.0
Lineer lrver sntertstttten
Unsartverraaamm
0.9
0.9
0.8
0.8
MPC
0.7
0.7
0.6
MyGausiia Linertisaa
0.6
UnearGarmma
0.5
0.5
6
8
2
Number of Latent Causal Variables
A large number of latent variables
Inverse Gaussian and Gamma noises
Figure 13: Performances of the proposed method on large number of latent variables and latent linear models with inverse Gaussian and inverse Gamma noises.
A.11 THE UNKNOWN NUMBER OF LATENT CAUSAL/NOISE VARIABLES
worth noting that most existing works require knowledge of the number of latent variables. However, this is not theoretical hurdle in our work. It is due to crucial step in results leveraging the identifiability findings from Sorrenson et al. (2020) to idenitify latent noise variables. The key insight in (Sorensoon et al. (2020) demonstrates that the dimension of the generating latent space can be recovered if latent noises are sampled from the two-parameter exponential family members. This assumption is consistent with assumption latent noise, defined in Eq, [ In other words, if the estimated latent space has a higher dimension than the generating latent space, some estimated latent variables may be related any generating latent variables and therefore encode only noise. More details can be found in Sorrenson et al. (2020).
A.12 MORE RESULTS AND DISCUSSION
In this section, we present additional experimental results conducted on synthetic data to demon- strate the effectiveness of the proposed method across scenarios involving a large number of latent variables as well as latent variables characterized by inverse Gaussian and inverse Gamma distribu- tions. Both scenarios present significant challenges in optimization. The performance these cases is depicted in Figure In the left subfigure of Figure [3] for ployGaussian, the presence of numer- ical instability and exponential growth in polynomial model terms complicates the recovery of latent polynomial causal models, especially with an increasing number of latent variables. Additionally, the reparameterization trick for inverse Gaussian and inverse Gamma distributions poses challenges in recovering latent linear models. Both cases warrant further exploration in future research.
22


Reviews:
[{"review_id": "fzkbkNKngp", "reviewer": "Reviewer_kugU", "summary": "The present manuscript is concerned with the extension of Liu et al. 2022's identifiability result to non-linear models with more general noise terms. The authors present as core contribution a theorem that allows for polynomial models and exponential family noise. A brief discussion on the violation of change of the exogenous terms is presented. Finally, an empirical section on non- and synthetic data sets.", "questions": "TL;DR: No questions.\n\nEven though I've checked the appendix, I've not done so in utmost detail, still, I'm confident in being able to guess what the authors actually mean or refer to in situations of uncertainty, therefore, no questions are derived on that end. Furthermore, the lack of discussions on the actual key insights, for making the transition to polynomials & the exponential family work, does not allow me to raise any further questions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "IMHO the paper's noteworthy strengths are limited to their idea rather than the execution, therefore, they should be considered (where applicable) as counterfactuals for the moment. Said \"potential\" strengths are considered one-by-one in the following list (the list is ordered in correspondence to the paper presentation):\n* Causal Representation Learning is an exciting and challenging avenue of research, naturally, extending prior results, especially the generalization sort of results, through precise characterization of key necessities is of great value. The authors place an effort in getting prior results to work for larger model classes such as polynomials.\n* An effort of presenting a self-enclosed treatise, covering both theory and empirics.\n* A discussion of partiality, as arguably most of the time in practice we work with approximations.\n* Although this should IMO not count as \"strength\", given that a large portions of papers within the community don't abide by this, but the fact that the authors present a clear and transparent communication of their assumptions.", "weaknesses": "The paper suffers from several disadvantages, ranging in importance from minor to more fundamental. The more fundamental ones are of major content/technical concern, and given that this review takes a content-centric approach, they weigh the most for the low scoring. The following list - again one-by-one - aims to provide specific pointers with improvement suggestions if applicable (please note, the list is unordered):\n* Liu et al. 2022 plays a central role in this work, since this work poses a direct successor in the form of an incremental improvement. While an incremental improvement can be of substantial nature, IMHO this work poses a Corollary-type of discussion over Liu et al. Furthermore, the presentation (especially in direct comparison to Liu et al.) does not provide any further justice to the present manuscript's proposed contribution. There is an abundance of repetition of the form \"we've extended it now\" accompanied by a lack of how alleged extension was achieved. Consulting the appendix did not resolve this major concern.\n* Kronecker product definition is not clear, not the usual definition e.g. $(a_1, a_2) \\times B = (a_1\\cdot B, a_2\\cdot B)$. \n* Although I'm confident in being able to guess what the authors refer to in their convoluted discussion on the data-generating process, it really does not help the fact that all shades of exogenous/latent terms (z, u, n) are not being explored thoroughly from first principles. Putting myself in the shoes of a reader who would be unfimiliar with the literature, I'd be tempted to expect that this portion would lead to major confusion.\n* Unusual, and IMHO absurd, number of assumptions paired with severity in some cases (e.g. bijectivity of $\\mathbf{f}$ is exploited oftentimes and arguably standard within the literature, although this should be questioned as well at some point, but existence of a suitable $\\mathbf{L}$ seems more tricky). This point shines especially in considerations of claims of appraisal made by the authors, such as \"[...] significantly bridg[ing] the divide between foundational theory and practical applications\", that seem to overestimate the presented contribution(s) and this, with all due to respect, in a debatably arrogant manner.\n* Corollary 3.2. seems to carry no value when considering that the whole paradigm of change of causal influence is based on the complete change assumption.\n\nFinally, following list provides concrete recommendations/pointers as to how the work might be improved such that its potential value for the community can be recognized/achieved. Again, this review is intended, respectfully, as constructive criticism and by no means as an attack to the authors. Sincerely, I hope that some of the below aspects can help:\n* Given the heavy reliance on Liu et al. 2022, the authors should consider actually saying what \"transitivity\" is within their introductory section and avoid using formulations that Liu et al. use in different contexts (e.g. \"absorbs\").\n* Please consider rewriting the introduction s.t. redundancy with Section 2 is being reduced. The extra space that is won through this is valuable presentation space, which avoids having to put otherwise more important results into the appendix.\n* Please consider rephrasing Fig.1 caption as not simply being a copy of the introductory text description of the Figure.\n* Please consider removing the pre-text at the beginning of Section 3, in favor of relevant discussions, as it arguably provides no additional value given all the text before (Abstract and Sec.1,2)\n* Please consider improving accessibility to the Figures by resolving the issues on the following two dimensions: (1) label/legend and general font sizes but also (2) proportions e.g. the choice of content being presented (e.g. Figure 6 and 7 do not convey valuable information w.r.t. this aspect).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUddseJaKf", "reviewer": "Reviewer_UsXb", "summary": "The papers deal with the identification of latent causal models from high-dimensional observations and provide results for the case of polynomial causal relationships, which extends over the prior results that were limited to linear causal relationships along with gaussian noise. They utilize the commonly used assumption of an exponential family distribution conditioned on surrogate variables for the noise variables, which helps them identify the noise variables up to permutation and scaling. Then they show how the identification guarantees on the noise variables can be used to get identification guarantees on the latent variables themselves using the assumption of polynomial functional relationships that show variation with the surrogate variable, similar to recent works leveraging interventional data for latent identification.", "questions": "I would mostly like a follow-up discussion on the concerns I raised in the Weaknesses section above. There are some other minor question mentioned below:\n\n* Is there any particular reason why the authors did not compare against several latent identification works that leverage interventional data? That would be a better choice of baselines as compared to Beta-VAE, which does not explicitly utilize the variation of latent variables with auxiliary variables. \n\n* It would be good to plot the F1 score along with SHD for the performance of latent causal discovery as well. The authors could maybe set a threshold to determine the latent causal structure from the functional relationships and compare that against the true latent causal graph.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written with a clear description of the various assumptions needed for the theoretical results, along with an extremely good proof sketch! Further, the discussion on prior work is quite easy to follow and places the contributions of the work in a good manner as compared to them.\n\n* The identification result provided in the paper uses fairly standard assumptions in the literature, but the application for the task of learning the latent causal relationships (not just the latent causal variables) is novel, and also somewhat significant as it is not limited only to the case of linear gaussian latent causal model.\n\n* The proposed approach is principled and the claims made in the paper are supported by experimentation over various benchmarks, however, the scale of the experiments is limited (refer to the Weaknesses section below for more details).", "weaknesses": "* My major concern with the work is regarding the technical contribution and significance of the proposed identification result. The proof uses common ideas of exponential family distribution [1] to obtain identification guarantees up to permutation and scaling for the noise variables. Further, the idea of using the changing distribution of latent variables identified up to polynomial mixing can be used to obtain permutation and scaling identification guarantees has been explored in prior works [2]. While I agree that the application of these ideas to the task of learning latent causal relationships is not trivial (as noted in the Strengths section above), I am not convinced by the significance of the proposed identification result. In terms of only the latent identification (not recovering causal relationships), the prior works that leverage interventional data can already provide permutation and scaling guarantees with much more general latent structures. For the task of learning the latent causal relationships, the permutation indeterminacy in the learned latents would pose a huge challenge which the authors circumvented by assuming the correct topological order. This makes it difficult for me to assess the importance of the proposed identification result. \n\n* While I appreciate the experimentation over several benchmarks, the scale (dimension of the latent variables) of the experiments is too small, with the maximum dimension of the latent space being 5. It would be nice to see the results with the proposed approach for larger latent dimensions ($d= 10$ or $d=20$ at least) for the synthetic benchmark. \n\nReferences:\n\n[1] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and\ngeneralized contrastive learning. In The 22nd International Conference on Artificial Intelligence\nand Statistics, pp. 859–868. PMLR, 2019.\n\n[2] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa-\ntion learning. In International Conference on Machine Learning, pp. 372–407. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDD3IEKHPl", "reviewer": "Reviewer_KC1H", "summary": "This paper proposes a novel theoretical result for causal representation learning, where the latent causal variables and structures are identified by exploiting the changes of causal influences across multiple environments. Built upon (Liu et al., 2022), the paper generalizes this previous result from linear Gaussian models to polynomial models with exponential family noise, as well as reducing the number of required environments. Identifiability results under different scenarios of changing coefficients were also discussed in detail. The paper also presents an empirical estimation method and validates it on synthetic and real-world data", "questions": "See my concerns in previous sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- In general the writing and presentation of the relatively clear and easy to understand.\n\n- The paper tackles an important and challenging problem of learning latent causal representations from observational data,  and makes significant theoretical contributions by extending the scope of latent causal models to nonlinear and non-Gaussian cases, and relaxing the number of required environments. \n- The paper did not stop after presenting their main results, but also provides in-depth discussions on partial identifiability results which is very useful and insightful. This will help provide more guidance under more realistic scenarios in practice.", "weaknesses": "I appreciate your feedback. Here's an improved version of the weaknesses section:\n\n1. **Latent structure identifiability**: While the paper makes significant strides in latent variable identifiability, it does not discuss the identifiability of the latent structure under the equivalence class. I understand that once the latent variable values are identified, the latent structure can be recovered trivially. However, given that the latent variables are only recovered up to an equivalence class, especially for partial identification case, it is unclear how would these affect the latent structure identification. This is a crucial aspect in causal representation learning, as understanding the underlying causal structure is often more important than identifying individual variables. The paper could benefit from a more thorough discussion or analysis on this aspect.\n\n2. **Functional assumption**: The necessity and expressiveness of the polynomial assumption are not adequately addressed in the paper. While it's understood that this assumption is needed for identifiability, it would be beneficial to have a more detailed discussion on why this specific form is chosen and how expressive it can be in capturing complex causal relationships. It would also be interesting to explore if there are other forms or assumptions that could achieve similar results.\n\n3. **Presentation**: Given that most of the assumptions have a strong graphical interpretation, the paper could benefit from adding more illustrations or visualizations to help readers better understand these assumptions and their implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FM7CNRP5P2", "reviewer": "Reviewer_22Xe", "summary": "UPDATE: The revised manuscript seems to address the concerns I had, though have not had the opportunity to re-review it. I have increased my score accordingly.\n\nThis paper approaches the problem of identifying the latent variable causal model from observed data, by leveraging changes in the causal mechanisms across environments. Identification here means finding the graph as well as the causal mechanisms. The causal mechanisms are generalized compared to previous work, allowing polynomial equations and exponential family noise distributions.", "questions": "\"Identifiability\": It is expected that it is impossible to exactly reconstruct the latent variables from the observed ones, e.g. due to the possibilities of permutation and scaling. So I would expect that your objective is identification up to certain transformations. However, a definition is not given and it remains unclear what you mean precisely by \"identifiability\" (and by \"unidentifiability\", rendering the claims of the corollaries unclear).\n\nChanges across environments: These are mentioned frequently in section 3.3, but it difficult to determine from the text what the precise meaning is: for a coefficient, is it (A) that there exist two environments such that the coefficient is different between those, or (B) that for each pair of distinct environments, the coefficient is different between them?\n\nTheorem 3.1 (i), \"measure zero (i.e., has at the most countable number of elements)\" - I assume you are referring to Lebesgue measure. In dimension 2 or higher, Lebesgue measure zero does not imply that the number of elements is countable. For instance, the set $\\\\{(x,0)| x \\in \\mathbb{R} \\\\}$ has Lebesgue measure zero, but uncountably many elements.\n\nTheorem 3.1 (iii) - What do you mean by \"takes up\"? Is it that the vectors $\\mathbf{u}_0$ etc are possible values that the random variable $\\mathbf{u}$ can take? (I expected that footnote 1 would say something about the distinction between random variables and their values, but I'm not sure that's what it says.)\n\nA \"DAG constraint\" is first mentioned in the proof sketch of the main theorem. It would be better form to introduce this assumption prior to the proof.\n\nStart of section 3, \"even if the causal relationships are nonlinear and noise distributions are non-Gaussian\" - isn't the linear Gaussian case the least likely to be identifiable?\n\nWhat is the $\\phi$ in the proof of Corollary 3.2?\n\nSection 4 (and elsewhere): To say that $z_1$ comes first in the causal order, write $z_1 \\prec z_2 \\prec \\ldots$.\n\nEquation (6): What does the $=\\mathcal{N}(\\ldots)$ mean? It contains unexplained notation, and $z_i$ doesn't appear except in indices.\n\nOther comments (e.g. typographical):\nPages 5 and 7: \"identifyable\" -> \"identifiable\"\nPage 5: \"polynomial propriety\" -> \"the property\"\nCorollary 3.2: This statement does not appear to follow from the preceding theorem, so the name \"corollary\" doesn't seem appropriate.\nAppendix A.4: \"support that\" -> \"suppose that\"\nAppendix A.4, \"proof of corollary A.3\" -> \"proof of corollary 3.2\"\nEnd of section 4: \"APPENDIX\" -> \"Appendix\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The latent causal representation learning problem is important, and I believe the extension proposed here is a significant improvement compared to previous work.\n\nThe writing is overall of a good quality. I appreciate the \"proof sketch\" and \"insights\" paragraphs following the main results.", "weaknesses": "The formal presentation needs work: most importantly, core concepts are ambiguous (see below for details). As a result, I am currently not confident about the soundness of the results. I hope these issues can be addressed with relatively little work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apzo3vTrjS", "reviewer": "Reviewer_YEeM", "summary": "This paper focuses on identifying a causal graph in the latent space. The paper extends previous methods as follows: 1) the dependencies between the latent variables can be polynomial (previously linear), 2) the noise of the latent variables can be in the exponencial family (previously Gaussian). The paper presents a theorem for the identifiability of such models, which in general is based on the principle of observing the system in multiple different environments. The number of environments required in the proof is smaller than previous proofs (2l+1 environments vs. the previous quadratic number). The paper also presents proofs for partial (un)identifiability, basically stating that for those parts of the model where the environments behave differently the latent variable can be learned, otherwise the latent variable is unidentifiable.\n\nMy overall initial view of this article is cautiously positive. However, I have multiple questions for clarifications.", "questions": "1) Could you comment on how to obtain the number of latent variables in practice? You are assuming it’s known in this paper, right?\n2) Condition (iii) in Thm 3.1 has u_{2l+1} although u’s exist only until 2l. Possible typo?\n3) Could you include additional intuition for conditions (iii) and (iv) of Thm 3.1?\n4) Proof sketch of Thm 3.1 says  “arises from the fact that additive noise models are identifiable”. If I remember correctly, additive linear Gaussian models are not identifiable? Could you clarify how this is compatible with the sentence?\n5) About Corollary 3.2: There is a condition: “if there is an unchanged coefficient in Eq 4 across environments, z is unidentifiable.”. What prevented such a condition from happening in Theorem 3.1?\n6) Second line of Equation (6): Where is z hidden on the RHS of the equation?\n7) Prior on lambda: you encure acyclicity by assuming lambda is lower-dimensional. Does this mean that you have to assume the causal order?\n8) MPC metric: correlation between what exactly?\n9) The last sentence of Discussion: “even with identifiability guarantees, it is still challenging to learn causal graphs in latent space”. It would have been interesting to explore how the number of samples affects this. Do I understand it correctly that the theory guarantees that in the limit of large number of samples the correct graph should be recovered?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper is very well written. The background is clearly explained and sufficient. Sketches of proofs are given for the theorems, which nicely give intuition. Mathematical parts (model, theorems, training) are rigorously presented. Overall, the presentation makes a nice trade-off between accessibility and rigorousness.\n2) The topic of latent causal modeling is challenging and the paper makes a few novel contributions in this challenging domain.\n3) The experiments are reasonably thorough and demonstrate and complement the theoretical developments in an intuitive way.", "weaknesses": "1) The paper relies quite a bit on a previous (unpublished) paper (Liu et al., 2022), which affects novelty. Overall, the paper makes a few contributions but none of these seems strikingly novel or signficant. However, the overall significance/novelty seems sufficient for acceptance in my opinion.\n2) The empirical experiments are somewhat artificial, and do not demonstrate real-world relevance (often the case in papers in this domain). The results are only shown for those setups where the method worked: LinearGamma, LinearBeta, PolyGamma, and not for the difficult cases: e.g. inverse Gamma etc. These could have been included in the Appendix at least, even if they didn’t work well (which the paper openly mentions).\n3) Some assumptions (additivity, knowledge of the number of latent variables) seem quite strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)