Unsupervised Learning of Causal Relationships from Unstructured Data
Authors: Marian Longa
Joao F. Henriques
Venue: Submitted to ICLR 2023 (2023)
Keywords: causality
deep learning
causal representation learning
unsupervised
VAE

Abstract: Endowing deep neural networks with the ability to reason about cause and effect would be an important step to make them more robust and interpretable. In this work we propose a variational framework that allows deep networks to learn latent variables and their causal relationships from unstructured data, with no supervision, or labeled interventions. Starting from an abstract Structural Equation Model (SEM), we show that maximizing its posterior probability yields a similar construction to a Variational Auto-Encoder (VAE), but with a structured prior coupled by non-linear equations. This prior represents an interpretable SEM with learnable parameters (such as a physical model or dependence structure), which can be fitted to data while simultaneously learning the latent variables. Unfortunately, computing KL-divergences with this non-linear prior is intractable. We show how linearizing arbitrary SEMs via back-propagation produces local non-isotropic Gaussian priors, for which the KL-divergences can be computed efficiently and differentiably. We propose two versions, one for IID data (such as images) which detects related causal variables within a sample, and one for non-IID data (such as video) which detects variables that are also related over time. Our proposal is complementary to causal discovery techniques, which assume given variables, and instead discovers both variables and their causal relationships. We experiment with recovering causal models from images, and learning temporal relations based on the Super Mario Bros videogame.

Under review as a conference paper at ICLR 2023
UNSUPERVISED LEARNING OF CAUSAL RELATION- SHIPS FROM UNSTRUCTURED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Endowing deep neural networks with the ability to reason about cause and effect would be an important step to make them more robust and interpretable. In this work we propose a variational framework that allows deep networks to learn latent variables and their causal relationships from unstructured data, with no supervision, or labeled interventions. Starting from an abstract Structural Equation Model (SEM), we show that maximizing its posterior probability yields a similar construction to Variational Auto-Encodes (VAE), but with structured prior coupled by non-linear equations. This prior represents an interpretable SEM with learnable parameters (such physical model or dependence structure), which can be fitted to data while simultaneously learning the latent variables. Unfortunately, computing KL-divergences with this non-linear prior is intractable. We show how linearizing arbitrary SEMs via back-propagation produces local non-isotropic Gaussian priors, for which the KL-divergences can be computed efficiently and differentiably. We propose two versions, one for IID data (such as images) which detects related causal variables within sample, and one for non-III data (such as video) which detects variables that are also related over time. Our proposal is complementary to causal discovery techniques, which assume given variables, and instead discovers both variables and their causal relationships. We experiment with recovering causal models from images, and learning temporal relations based on the Super Mario Bros videogame.
INTRODUCTION
Human reasoning and decision-making is often underpinned by cause and effect: we take actions to achieve desired effect, or reason that events would have happened differently had we acted a certain way if conditions had been different. Similarly, scientific inquiry uses the same tools, albeit more formalized, to build knowledge about the world and how our society can affect it (Popper) 1962). When building algorithms that automatically build statistical models of the world, is common in machine learning practice, it would then be desirable to imbue them with similar inductive priors about cause and effect (Glymour al. 2016). In addition being models which only characterize the observational distribution (Peters et al. 2017), they would allow reasoning about changing conditions outside the observed distribution (c.g. counterfactual reasoning). They would also allow communicating their inner workings more effectively - allowing us to ask "why"" given conclusion was reached, much in the same way that we do in scientific communication.
robust than statistical
Despite still being actively researched, there is now a mature body of work on understanding whether two or more variables are related as cause and effect (Peters et al. 2017). Many techniques assume that the variables are given, and concern themselves with finding re relationship between them (Spirtes) Glymour 1991 Chickering 2003] Lorch et al. (02T). On the other hand, an advantage of modern deep neural networks is that they learn intermediate representations that do not have to be manually labeled (Yosinski et al. 2015),, and effective models can be trained without supervision (Kingma & Welling 2014). An important question then arises: can a deep network simultaneously discover latent variables in the data and establish cause- cause-effect relationships between them?
We focus learning Additive Noise Models (ANM) with Gaussian noise, which are identifiable (i.e. causal directions are distinguishable) as long as the functions relating the variables of interest are not linear (Hoyer al. 2008). This model fits well variational learning framework, and we able

---

Under review as a conference paper at ICLR 2023
to derive analogue of Variational Auto-Encoder (VAE) (Kingma & Welling 2014) where the prior, rather than being an uninformative Gaussian, corresponds exactly to the ANM. When the ANM is linear with Gaussian noise, the joint probability of the variables also becomes Gaussian, and it is easy to perform variational inference. The dependencies between variables will then be expressed in the covariance matrix's sparsity structure. However, mentioned earlier make the causal directions identifiable the model cannot be linear (Hoyer et al. 2008). We resolve this difficulty by learning models that are locally linear, but globally non-linear. This approach affords the full generality of con-linear ANM, with the simplicity of variational inference on Gaussian models.
In summary, our contributions are:
A rigorous derivation of the variational Evidence Lower Bound (ELBO) of an Additive Noise Model (ANM), allowing efficient inference of Structural Equation Models (SEM) with deep networks.
A linearization method leveraging automatic differentiation to construct a local Gaussian approxi- mation of arbitrary non-linear ANMs.
A lemporally-aware specialization of the causal ANM that encodes causal directions implicit in the arrow-of- of-t-oeemme and suitable for high- high-dimensional time series data such as video.
Experiments demonstrating that the proposed method is able to fit latent variables with a dependence structure in high-dimensional data, namely a synthetic image dataset and video game based data.
RELATED WORK
Our work lies the intersection of causality, variational inference, representation learning, and high-dimensional unstructured input domains.
Causal inference deals with determining the causes and effects from data. Causal discovery methods generally focus on recovering the causal graph responsible for generating the observed data, e.g. Spirtes & Glymour (1991): Chickering (2003) (for overview of methods in see Peters et al. (017)) However, these methods are largely applied to structured datasets such as medical (Brooks-
1992 [aahs al. 2005 Louizos et al. (007)) or economics data LaLonde (1986) where the observed variables are provided by domain specialists. In contrast, we focus on unstructured data where the variables are not provided a priori.
Variational inference is a way of performing inference by solving an optimisation problem. A popular instance is the Variational a useful latent representation of the data by encoding and decoding it back. Traditionally the VAE prior is assumed to be an isotropic Gaussian distribution and the aim is to extract independent latent variables such as in the 3-VAE (Higgins et al.) (2016b) and Factor (Kim & Mnih 201 8). There are works which use hierarchical priors such as iteratively conditioning each variable on its preceding variable in the Ladder- VAE (Sønderby et al. 2016) and conditioning each variable all its predecessors in NVAE (Vahdat & Kautz] (0200) and VDVAE (Child 202I). We also use prior conditioning each variable its predecessors but this comes natural consequence of basing our prior on a structural equation model (SEM).
(VAE) (Kingma & Welling, 2014) which aims extract
Recently there has been a growing interest in representation learning based causal principles. For instance, the Causal (Yang et al. 2021 learns independent latent variables which are then composed to form causal relationships, however they only consider linear relationships between variables. Other works use different approaches to VAËS for causal learning such as the CausalGAN (Kocaoglu et al.) (2188) which use generative adversarial networks. Yet another line of work focuses on modelling object dynamics from video such as Li et al.) (2020). however they use specialised modules for detecting keypoints and future prediction. Another line of work uses graph neural networks to infer an interaction graph such as Kipf et al. (2018); Löwe et al. (2022), but they do not deal with image or video data. Lippe et al. (2022) focus on causal learning with the knowledge of assume no such knowledge. Another line of work such as Lachapelle et al.
interventions, whereas (2022) and the iVAE (Khemakhem et al.) 2020) uses non-linear Independent Component Analysis theory. Locatello et al. (2020) explore using small number of labeled examples for learning. Walker ct al. 2021) use VQ- VAE for video future prediction using a hierarchical prior but do not focus on causal relationships.
2

---

Under review as a conference paper at ICLR 2023
3 BACKGROUND
In this section, we will give self-contained overview of several results from variational and causal inference that we build upon. While they not new, bringing them together under one formulation offers new insights and challenges, which we solve in sec.
Our goal is to fit distribution p(z), defined over input space & E R to an empirical distribution p(z) composed of finite samples, by choosing the optimal p E P out of a set of candidate distributions P (e.g. parameterized by neural network). We do this by minimizing their KL-divergence D₂ computed over x, or equivalently maximizing the expected log-likelihood of p over the dataset pi:
p' arg min D2 ()(())))l)((zr) arg max Be)p()) [I0 p(x)]] pEP
pEP
We now introduce set of latent variables IRd, which can marginalize compute
p(æ) p(=)p(xl2z)d=
in terms of conditional distribution p(r) =) and "prior" distribution over latents (=z). In standard
VAE, this prior is an isotropic Gaussian distribution (Kingma & 2e1ling|2014), while other structured priors are possible (Sønderby al., [016]] Vahdat & Kautz, [020 Tomezak & Welling 2018). In this
work, however, we will define it as Structural Equation Model (SEM) (Pearl 2009) (section 3.2
3.1 VARIATIONAL INFERENCE
We can now apply standard tools of variational inference (Bishop 2006 Ch. 10) eq. and replace
the intractable marginalization (eq. ( with an optimization of q E Q over a variational family of
distributions Q (in essence, training an additional neural network q). Eq. when marginalized (eq.
2 is equivalent to (Kingma & Welling 2014):
p- = arg max
E 2-ag(=
[In[[=])]] - D2 (1(=)=)ll((2))
PEP. xoup(x))
The first term in eq. amounts to a reconstruction error, and the second term matches the latent variables the prior. In practice, q(=|<x) and p(x|2) in eq. Jrre often defined Gaussian distributions parameterized by neural networks. These are the functions (2(x)) E R and Ex(x)) E dxdd for the encoder q, and (2) E Rd and (p(2)= E Id for the decoder p. which parameterize the means and covariances of the distributions:
(4) The KL-divergence between prior p(z) and variational posterior (=]xx) can be computed in closed form when both are Gaussians. Note that the encoder usually outputs diagonal covariance matrix (motivated by the fact that an isotropic prior is also diagonal), but here we allow it to output full covariances Egy which will be important later (sec. 4
<(=|>²) (=)m,(()), Eg(x))),
p(x|=) = -N(xl1 di)g(ap(=))).
3.2 STRUCTURAL EQUATION MODEL
We now consider set variables y, which have dependency structure defined by a directed acyclic graph (DAG) G (i.e. there is an edge (i - f) G if y, is required compute yy). Additionally, the generation process for each variable yi can be described by a sequence of non-linear equations:
M f. (1) + n4.
where f. (1 + R. is an arbitrary deterministic function, (i) denotes the indices of parent nodes of i in G. and n, [N,,,(, of) an independent zero-mean noise variable, assumed to be Gaussian. This represents a Structural Equation Model (SEM), more specifically an Additive Noise Model (ANM) (Peters et al. 2017 Ch. 4.1.4) with Gaussian noise, which has the joint probability:
pvy) II Pe(s ())()) = II NV: - (i 10, of).
Practical methods for causal learning with ANMs typically assume the variables y are observed, and are concerned with recovering the true causal graph G (and sometimes the functions f) that generated the data (Hoyer al. 2008 Peters et al. 2017)
3

---

Under review as a conference paper at ICLR 2023
3.2.1 SCORE-BASER MODEL SELECTION
The formulation so far has assumed that the causal graph G, which encodes all the dependencies between variables, is given. We can, however, make use of result by Nowzohour & Bühlmann (2016) that shows that a penalized likelihood score can be used to select between different models Pg(y) (each fit to empirical distribution p(y)), with each assuming different graph G' g (e.g. out of all graphs with up to d nodes). We thus select the graph that has the maximum score:
1
|
G = arg max Gcg
In (v)
In
gr~A(y)
with the number of samples, and |G] the number of edges of G. This method finds the true causal graph when the causal dependencies in the ANM are all non-linear (Nowzohour & Bühlmann (016))
4
METHOD
4.1
VARIATIONAL INFERENCE WITH A CAUSAL PRIOR
The previous exposition suggests a natural way simultaneously learn latent variables and fit causal model: use the ANM (eq. (6) as the prior in the variational optimization problem (eq. 33. by setting y = 2. This amounts to assuming that the latent variables z have dependency structure defined by a DAG G, and are generated sequentially by application of the non-linear functions fa corrupted by Gaussian noise nd, Model selection (i.e. finding the optimal graph G) can then be done by the score-based search of sec. 3.2. Note that this model reduces to VAE for DAG with no edges and thus null functions fi:
Vi: pa(i)= = 24 = n2 (reduction to standard VAE)
(8)
At a high level, the same tools used to train VAE should be applicable in this new setting. However, the ANM is only identifiable (i.e. the true causal directions can be recovered) if the f, are non- linear (Hoyer al. 2008). and this makes computing the KL-divergence in eq. B iintractable, since no longer defined in closed form. We will resolve this difficulty by local linearization (sec. (.4). although the model will still be globally non-linear to ensure identifiability.
One major difference from the exposition by Nowzohour and Bühlmann is that they propose using non-parametrie fitting method over known variables x, while we want to simultaneously recover the causal structure and latent variables z (e.g. unknown parameters of objects depicted in images), which must be estimated from inputs x (e.g. raw pixels). Another difference is that their procedure is computationally expensive, as it entails enumerating all graphs G E G explicitly. However, given the expressiveness of deep neural network models, in sec. 4.2 we will show how we can replace this search with a simpler model fitting procedure.
4.2 MAXIMAL DAG
For the purposes of fitting the model G and observed data x, we will consider a simplification where we take all ancestors of a variable 2; to be its parents pa(i), i.e. we replace G' with the full
DAG with edges (i - < j, i, I,...,d) d). The following proposition shows that this will be able to model any underlying SEM, although some of the independence relations will have to be modeled by the learnable edge probabilities described in sec. 4.5
Proposition 1. The set of all SEMs = {(2 = folmma (0)) VG, f) with arbitrary DAGs G
and probabilistic functions f is contained within the set S {(2, = fi(2)....-1))d. [Vf), up to reorderings of the variables z.
Proof. See Appendix A
Prop. I says that, as long as the function class of f is expressive enough (as usually the case with deep networks (Yosinski et al. 2015),, we can find equivalent SEM using fixed maximal DAG G with edges {i +
J. J,J = 1.....d) d). This simplifies the exposition in the following sections and provides justification for the approximation in sec. 4.5
4

---

Under review as a conference paper at ICLR 2023
4.3 PRIOR PROBABILITY FOR LINEAR SEMS
In the simple case when all functions fi of the ANM are linear, the resulting joint probability must be Gaussian, as is a linear combination of independent Gaussian noise variables n, (eq. recalling that y = 2). However, we need to compute its precise form for general linear ANMs.
Theorem 2. Consider a linear ANM defined as 2, = of 21....--1 + by + n4, with a, E ER'11, b, e R and ni ~ [Nnn,(0, of) for i = 1,.... d. Missing edges the causal graph can be represented as zeros in ag Then the ANM's s joint probability is given by p(z) [2z u1 2) with
2 II A
2
2
b,
==
II A lidd
diag (a
II A
(9)
v=dd
1 1(i-1)x(i-1)
O1-1 O (1-1)x(d-1))
81
A; A
Ou ((a-sxidd-))
b
(a--i)x(i-1)
Oa-
ba
where Ixck denotes an identity matrix, while Oş and Oxxt denote a zero column vector and matrix.
Proof. See Appendix C
L
The process in Theorem 2 can be interpreted as form mean and covariance propagation: at each stage the ANM applies linear transformation to the mean and covariance from the previous stage. The matrix Ii A; is lower-triangular matrix representing the edge strengths from parent to child nodes in G. obtained from the SEM. This linear transformation is then applied to the SEM biases to obtain the total mean μa, as well as the SEM noise variances of to obtain the total covariance L. For identifiability, however, we must generalize this process to non-linear ANMs, which we do next.
4.4 PRIOR PROBABILITY FOR NON-LINEAR SEMS
Our approach to deal with non-linear ANMs is to linearize them around pivot point 2". Due to Taylor's 's theorem this approximation will be accurate for a sufficiently small neighborhood (Nocedal & Wright 1999 Ch. 1). The ANM's joint probability p(z) will then be Gaussian by Theorem 2
Theorem 3. The best linear approximation (in the least-squares sense) of an ANM 2; fe( + n4 (eq. 5 with missing edges in the causal graph corresponding to ignored in- puts in fe). around a pivot point z², is given by eq. (Theorem 2 with with
(f:(21....11))
04 021,..i-11
and b, Si( + 0g ii.....1
(10)
where oी 111, =1 =P.....
f (= + n, and n, NN0, of) with learnable os.
Proof. See Appendix D
The advantage afforded by Theorem is that the ANM's joint probability p(2) is locally Gaussian, and we can compute its parameters by mean and covariance propagation (eq. 9.. Another advantage is that, in the context of training deep networks, one can use automatic differentiation prock-rropagation) to compute eq. 10 for arbitrary functions ft. including very expressive ones such as multi-layer perceptrons (MLP). Since this locally-linear SEM is represented an explicit Gaussian distribution, the objective's 's KL-divergence (eq. (3) can be computed in closed form (Kingma & Welling, 2014). We can then obtain the full prior distribution by sampling many pivot points 20 by ancestral sampling according to Theorem
4.5 GRAPH SEARCH VIA SPARSITY
To efficiently search for the graphs in eq. 7 we use Proposition I to justify using a fixed maximal graph GO to learn a single SEM (z(2) (where we make explicit the dependence of p(2) on the learned causal functions fi, c.f. eq. 5). This is in contrast the previously-proposs enumerating all graphs E G and learning a different Pz,a(z) for each G (Nowzohour & Bühlmann 2016). We define the graph of dependencies:
technique of
5

---

Under review as a conference paper at ICLR 2023
VAE (3 I)
VAE (5 4)
Causal-prior VAE
Prior
47
4
Observed
Figure 1: Samples from the observed and prior distributions for different VAEs, showing that the distributions match only for the 3 = and the causal-prior VAEs. Only the causal-prior VAE recovers the underlying causal structure that relates the latent variables, parabola.
Definition 4. We define the implicit dependency graph G(f) as the graph with d nodes, and each edge i + j exists if fj fz(21....j-1) depends on its ith input.
We can then define the overall objective (from eq. and eq. 3 as
G(DI In
p arg max
Σ
E 2-~9(=12)
[np(x|=)] D2 (1()a))l)s)))
(11)
PEO. B=)(
To approximate the [G(]) term that counts the edges of the graph G(J), we introduce edge prob- abilities Pi+j denoting the probability that an edge from node i to j exists, which allows us to write [G(f) = Ej.cs Pi-j (i.e. for binary probabilities, this recovers the exact edge count). We define each pı-+j as binary Gumbel-Softmax distribution (Jang et al. (2017), which is suitable for gradient-based optimization. The edge probabilities mask the inputs of the SEM's functions (Z1....,Zz_)) as (P (...---.......11111---.... 2)-1-133-11) During model evaluation we can sample binary edge probabilities Pi-tj E {0, 1}, producing a discrete graph G.
We can thus use standard stochastic gradient methods to optimize eq. IT and obtain an encoder g. decoder p. and SEM defined by f and pi-+j: the latter of which models well-defined causal directions (as long as the underlying functions are non-linear) and reflects independence relations in the data.
4.6 IDENTIFIABILITY
In Theorem we demonstrate the identifiability of the proposed model up to some unavoidable indeterminacies. The only transformations that can be implemented by the learned deep networks (encoder, decoder and SEM) and can result in an identical objective value to the optimal model are reorderings of the latent variables, and as well as shifts and orthogonal transformations of the input (implemented by both the encoder at the input, and decoder the output). The conditions required to achieve this result depend common features of standard deep networks, namely encoder and decoder composed of ReLUs and linear operators, batch normalization in the outputs of the SEM f, and SEM with fixed scale, such as the quadratic used in experiments. The proof is given in Appendix B
5 EXPERIMENTS
In this section we validate method on two experiments: the first one learning atemporal variables and the second one learning time-varying variables. the atemporal experiment (Section 5.1 we show that our method correctly recovers the positions of shape placed on parabola given fixed quadratic prior. the time-varying experiment (Section (.2) we show that our method correctly recovers the positions of moving character and their relationships over time, using learnable linear prior.
5.1 LEARNING AT CAUSAL RELATIONS
In this section we demonstrate the method's s ability to learn causal relationships that are not time dependent. We dataset of images of oval shapes different rotations and scales, placed at

---

Under review as a conference paper at ICLR 2023
44
4
VAB (3 1)
VAE (3 4)
0.03
a 0.04
a=0.04
a-0.77
0.08 a-
d=0.08 A
a=0.13
0= 0.15
0.0.48
0.0.96
2
2>
24
Causal-prior VAE
a-0.10
a=0.22 a-1.6 d-023
Figure 2: Latent response to the object's position for different VAEs (higher values are brighter). The 3 4 and B 1 VAEs have entangled responses (although it is less entangled for B = 1), while the causal-prior VAE correctly disentangles the object's horizontal and vertical positions.
positions that follow a parabolic are y = x² + n (with noise n)). We thus have a causal graph with a single edge r + y. We train our causal-prior VAE with 3-layer MLP and parabolic prior (see Appendix B for details), and use the L2 loss between each image and its prediction from the decoder as the reconstruction error (see Appendix ??). We also train high-8 and low-3 VAEs for comparison (where 3 denotes the multiplicative factor of the KL divergence as in (Higgins et al. )016a)).
Latent space visualisation. In Figure 1 we show the prior (blue) and the predicted distribution (red) for the low-ß VAE, the high-3 VAE, and our causal VAE. For the high-8 VAE (center) the two distributions match, but the latent variables are entangled (i.e. both follow an isotropic Gaussian distribution), and so do not reflect the underlying parabolic relationship between the variables that generate the data (y = x² + n). For the low-3 VAE (left), although the observed variables recover the underlying parabolic shape (red), up to vertical reflection of the coordinates, this distribution does not match the prior at all (blue), since it is constrained to be Gaussian. As such, this generative model p(x) (eq. 2 cannot generate samples that respect the causal relationships in the empirical data distribution. For our causal-prior VAE (right), the two distributions match and both follow the underlying noisy parabolic equation. Our model can thus be used as a generator p(z) that reproduces the true causal structure of the data, a capability that we will explore next in more detail.
Latent response to position. Figure shows for different VAEs the value of each latent variable as the position of the shape in the input image is varied, averaged over different rotations and scales. The variables are ordered by their standard deviation (o). The goal assess how sensitive each variable is to the depicted shape's s position. For the causal-prior VAE, the variable 21 clearly corresponds to the shape's s position (with values linearly increasing towards the left), and variable 22 to the y position (with values increasing towards the top). while other variables show minimal response. For the high-8 VAE the position is entangled across several variables; for the low-3 VAE the variables 24 and #1 correspond approximately to shape's x and y position, but there is still significant entanglement with the other variables, which are also highly sensitive to the shape's s position.
Latent noise traversal. In the first and third columns of Figure 3 we show grids of images decoded by each method, obtained by taking one sample and varying the noise n, of the two latent variables 2 (see eq. that best correspond to horizontal and vertical position. In the first column, decoded images are color-coded by their horizontal position. Since the positions of shapes in this grid are difficult compare, the second column shows the same images but superimposed, which reveals the underlying parabolic structure of the data. The third column shows the same grid of decoded images, color-coded by vertical position, while the fourth shows them superimposed. While all methods seem to generate images following roughly the parabolic data distribution, we can observe that in both 3-VAE configurations the shapes' positions are entangled (i.e. vary jointly) with their rotation and scale, while the same is not true for the proposed causal- -prior VAE. It is interesting to note that in the later case, the noise ni (corresponding to variable 21) does not map directly to the shape's 's vertical position (as observed for the 8- VAE), but rather expresses its offset from the parabola, according to
21 z2 + n1. In the bottom-right panel, the red tint of images generated from positive noise offsets
(n1 0) is visible above the parabola, and the blue tint of negative offsets is visible below.
5.2 LEARNING TEMPORAL CAUSAL RELATIONS
We now evaluate our method on temporal data, namely short videos. We apply the encoder and decoder independently each frame, and append an encoded background frame to the the decoder's
7

---

Under review as a conference paper at ICLR 2023
430
VAB (3
+000
+20
-10
-10
20
20
-10-22-10+500 10HH0HLL-0E
VAE (3 = 4)
100
130
-10
-10
-20
-10
-30
-10
Causal- Causal-prior prioorro VAE
2:400
-55
Figure 3: Reconstructed images (as 2D grid and as superimposed images) as latent noise values are traversed at regular intervals. For the 3 = and / = 4 VAEs the depicted shape's position is entangled with its rotation and scale, while for the causal-prior VAE the horizontal position and vertical offset from the parabola are disentangled correctly.
input (to allow the latent variables to ignore background/ime--nvariine appearance). The SEM only contains edges in the direction the arrow of time (see Appendix for details). We create a dataset based on the Super Mario Bros video game depicting object moving with linear motion in different directions, at different speeds, and on varied backgrounds. Three frames are observed, with the object's coordinates (xz, 34) for E {1,2) sampled uniformly from [7, 12], and the third obeying 21/2 V1- We train our method with linear prior, and otherwise similarly (see Appendix
Iy 20 31,
F for details) and select the sparsest graph that still achieves the best reconstruction accuracy. We use the reconstruction error given by Σ L2(2,, 2.) + L2 L2(23, i where denotes a frame at time t, 2, denotes a prediction made using the decoder p(fxl=e) and a/ denotes prediction made using the decoder p(izg If (21, 22)) (see Appendix ?? for details). For quantitative evaluation of learned graphs please refer to Appendix G
Sec. 5.1
Learning causal graph. Figure 4 shows the graph learned by the causal-prior VAE, with the bot- tom row (21, 22, 23, 24) corresponding latent variables at time 1, the middle row (25, 26, 27, 28) corresponding to = 2 and the top row (z9, 210: ø11, #12) corresponding to / 3. Learned edges are shown in black and missing ones in beige; learned functions fi are represented as number (multi- plicative factor) next to the corresponding edge, since they are linear in this case. The graph shows that each component of the object's 's 2D position at time = 3 (expressed as (=10+ 211)) depends on its 2D positions at times = 1 (22, 23) and t = 2 (#g, 27), following the model and 211 = fu1 (2) = 2.227 1.12s, which matches the data generation process up to a scale factor.
fio(z) 2.32g - 1.122
Latent response to position. Figure 5 shows the response of each latent variable as the object's horizontal and vertical position varied the input. The rows correspond variables from times t = 3, 2, respectively. We can see that 22, 26, 210 encode the object's 's position along the bottom-right to lef-left diagonal at different moments in time, while 23, 27, 211 encode the position along the orthogonal direction. The remaining variables do not exhibit any significant response to position. This shows the variables correctly match the data generation process up to rotation.
Interventions visualisation. Once a SEM relating the latent variables has been learned, we can perform interventions some of its variables. In Figure 6 we encode randomly sampled reference
8

---

Under review as a conference paper at ICLR 2023
Zg
211
Z12
12
Z5
Z6
28
21
Figure 4: Learned graph edges and mechanisms relating the latent variables, recovering the data generation process up scaling factor.
Figure 5: Latent response of variables to the ob- ject's horizontal and vertical position, recovering the data generation process up to rotation.
Z2
Z3
Z6
Z7
Z10
Z11
other
X1
X2
83
84
X5
-3.0
F
1.5
+0.0
F
+1.5
+3.0
Fn
2=2
Figure 6: Intervention results showing that inter- vening each variable propagates to its descen- dants through the correct mechanism.
Figure 7: Extrapolation results obtained by itera- tively passing the encoded variables through the learned SEM to predict their future values.
clip from the dataset and separately intervene by assigning different values (rows) to variables ..., 212 (columns), propagating the changes to their child variables using the SEM, and decoding the results. The results are decoded using a black background and the resulting 3-frame decoded
sample is combined into single frame by colouring the t = decoded image by red, the = 2 immae
by white, and t by green. We observe that intervening the variables for 1 (22. 23) changes both the object's position in L = 1 (red, columns 1,2) and = 3 (green, columns 1,2), following the SEM. Intervening
the variables for / (ze; 27) changes the object's = 2 position (white, columns 3,4) and = 3 position (green, columns 3,4); while intervening on the variables for t = 3 position (green, columns 5,6), which reflects the fact that other
(210, 211) changes only its positions (in the past) do not depend on it. Intervening on other variables has no effect (last column). This confirms that the model has learned the correct temporal causal structure, as each intervention affects the right variables.
Extrapolation visualisation. Having learned SEM relating those at time to the variables at times (t 1-,,,- 1,t-2) now makes it possible to extrapolate the variable values into the future. In Figure 7 we start with 3 samples (rows) of consecutive frames from the dataset 31, T2, encode them to obtain the = 1, 2 latent variables (æ1, 28), and iteratively pass them through the SEM to compute the / latent variables (2g, ..., 212), decoding these into 3.. We now repeat this process by taking 25: 212 as the f 1,2 variables and use the SEM to predict the = 3 variables which we decode into 2. and similarly for 25. thus obtaining predictions for time steps into the future. The future predictions confirm the knowledge learned in the SEM that the object moves linearly and can be used to predict future frames accurately.

---

Under review as a conference paper at ICLR 2023
6 CONCLUSIONS
In this work we proposed general model that naturally extends the variational learning framework to learn non-linear Structural Equation Model (SEM) as the prior, which enables causal learning to be performed SEMs while using Gaussian variables we proposed a fully differentiable method that locally linearises the SEM to obtain Docally-Guussian distribution. Furthermore, graphs as a joint continuous optimization over non-linear causal functions f- The proposed method shows promise scale to nigh-dimensional data and moderately complex SEMs, however future work should explore more large-scale data such as long videos and other input modalities.
perceptual modalities such images and video. To reconcile the non linearity of
relaxed the search over causal
REFERENCES
Jørgen Bang-Jensen and Gregory Z. Gutin. Digraphs: Theory, Algorithms, and Applications. Springer London, 2009. ISBN 978-1 1-84800-9997-4
Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN 978-0- 387-31073-2.
Brooks-Gunn, F. Liaw, and P. K. Klebanov. Effects of early intervention on cognitive function of low birth weight preterm infants. The Journal of Pediatrics, 120(3):350-359, March 1992. ISSN 0022-3476. doi: 141101660022-334660555888333
David Maxwell Chickering. Optimal structure identification with greedy search. The Journal of Machine Leurning Research, 3(mul)):507-554, March 2003. ISSN 1532-4435. doi: 10.1162) 15324430332189771 URL
Rewon Child. Very deep (vae)s generalize autoregressive models can outperform them on images. In International Conference Learning Representations, 2021. URL httl:ttps////pereeeeee
net feet/fosae worummrrmmi.. iic.ummiiiiii -i-.unidddRRRRVVDD55 3LRXCV6DBEJ
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley Sons, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pp. 1026-1034, 2015.
Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, and Alexander Lerchner. Early Visual Concept Learning with Unsupervised Deep Learning. Technical Report arXiv: 005766576607, arXiv, September 2016a. URL http::////// orq/ abs 6 05579 arXiv:1606.0557) [cs, b-o, stat] type: article.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. November 2016b.
ptatpp:://penneriev.ee...
Forum?1d=Sy2fzul
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL httphhttt:///ppeett wwp..nippp....m..........m cc/
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gufbex- In International Conference on Learning Representations, 2017. URL httl//ttt///peeeeiiiiii
10

---

Under review as a conference paper at ICLR 2023
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In Silvia Chiappa and Roberto Calandra (eds.), Pro- ceedings of the Twenty Third International Conference volume 108 of Proceedings of Machine Learning Research, pp. 2207-2217. PMLR, 26-28 Aug 2020. Uttpttss:: pttttss://proceediiie ngs ml www.s/rsssvvvv...
Artificial Intelligence and Statistics,
/ http:///w000//eekaaaaaaaaaaa 0a html
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising.
Proceedings of the 35th
International Conference Machine Learning, pp.
PMLR, July 2018. URL ISSN: 2640-3498.
mlr. press pentess/v8le /v80 v/0 //immlmmml 8b. html
Diederik P. Kingma and Max Welling. Auto- Auto-Encoding Variational Bayes. Technical Re-
port 1rXiv:1312.6114, arXiv, May 2014. URL httpp://axxv..rrtt abs abs//bls..../// 6114
arXiv:1312.6114 [cs, stat] type: article.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learming Research, pp. 2688-2697. PMLR, 10-15 Jul 2018. mlr. wwwppppeessssseeeeee
html
Murat Kocaoglu, Christopher Snyder. Alexandros G. Dimakis, and Sriram Vishwanath. CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training. February 2018. URL
hitts:://ppeneellews wwt.:e/wwwwww..eeoeeeeeeeeeeeeee Fot.ee//ooo.mm............ -d=
Sebastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi LE PRIOL, Alexandre Lacoste, and Simon Lacoste-Julien Disentanglement via mechanism sparsity regularization: new principle for nonlinear ICA. In First Conference Causal Learning and Reasoning, 2022.
URL http::///gaaaaeeeeeeeee/e//c/ces
net for.m:id-cHsseee...e
Robert J. LaLonde. Evaluating the Econometric Evaluations of Training Programs with Experimental Data. The American Economic Review, 76(4):604 620, 1986. ISSN 0002-8282. URLhttps // //www. jstor. wwtpooogg///ooo..o Sastaaale/1805
Publisher: American Economic Association.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal Discovery in Physical Systems from Videos. In Advances in Neural In- formation Processing Systems, volume 33, pp. 9180-9192. Curran Associates, Inc.. 2020.
URL htttps:/proceeednngee hew.:ro.eeeeiiinn...r..ppppp.ppp wttpiips..ccppeeeeeee cct/papper/222//aash 6822557750444401188d022222222223
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves.
iCITRIS: Causal representation learning for instantaneous temporal effects. In UA/ 2022 Workshop on Causal Representation Learning, 2022. URL https::/openrevieeee net ffr.rorrrmdd
xeDKK2s21171
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Confer- ence on Learning Representations, 2020. URL htttps://operretssss wtw..etfffformmidec forum?id=
net/
SygagpEKиB
Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf. and Andreas Krause. DiBS: Differentiable Bayesian Structure Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 24111-24123. Curran Associates, Inc., 2021. URL htt.:/ttpss/pprroeednss.ssome .htm)
cc/
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal Effect Inference with Deep Latent-Variable Models. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ppoceeecinccceeoo http://ppooceeeinnn....e..epppp http/pppe/222//hasl1 B4bSbdde6deaBaafffeoo 9cd.eeddddfccceeeeeeeeee S.sS5eeeleeBBBoooooco66622222223 /
Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Learning to infer causal graphs from time-series data. In Bernhard Schölkopf, Caroline Uhler, and Kun

---

Under review as a conference paper at ICLR 2023
Zhang (eds.), Proceedings of the First Conference Causal Learning and Reasoning. volume 177 of Proceedings of Machine Learning Research, pp- PMLR, 11-13 Apr 2022. URL
hnttps://procee wwwpiann................. 1.a...preea/o77//owwe22222 hhtmmma...mmm
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Christopher Nowzohour and Peter Bühlmann. Score-based causal learning in additive noise models.
Statistics, 50(3):471-4855 2016.
Judea Pearl. Causality, Cambridge University Press, Cambridge, 2 edition, 2009. ISBN 978-0-521- 89560-6. doi: 10. 10.1017/CB097805111000 1803161. URL books https//cccaallltt///44444442 XTC.C/440......TCCCCCCCCCCCCCCCC 0CH227444CCDAEES5=
/www. wwwp://wwwwww...ooooooooo
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations
and Learning Algorithms. Adaptive Computation and Machine Learning series. MIT Press, Cambridge, MA, USA, November 2017. ISBN 978-0-262-03731-4
Sir Karl Raimund Popper. Conjectures and refutations: The growth scientific knowledge. 1962.
Karen Sachs, Omar Perez, Dana Pe' Pe'er, er, Douglas A. Lauffenburger, and Garry P. Nolan. Causal Protein- Signaling Networks Derived from Multiparameter Single-Cell Data. Science, 308(5721):523- 529, April 2005. doi: 10. 1061ie2// 105809, /RLhttps: /wwwww/w..... /twpw//wwwncce..eee/..oowwwe www.sccieeccceeeeee..ce org/doi/ Publisher: American Association for the Advancement
Full/10.1126/se 11l.0.112//ccceee www.eeeee.15000 1105809 of Science.
Peter Spirtes and Clark Glymour. An Algorithm for Fast Recovery of Sparse Causal Graphs. Social Science Computer Review, 9(1): April 1991. ISSN 0894-4393, doi: 10.1177/ 01944493310090000
URL
Publisher: SAGE Publications Inc.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL //tttpss:/ppppr..st .nps https/sccppppppppp..//e// 6/hash/ 6aae07dcb33ec:b1ee c1 4d Ccbda Attt
Jakub Tomezak and Max Welling. VAE with VampPrior. In Proceedings the Twenty-First International Conference on Artificial Intelligence and Statistics, pp. 1214-1223 PMLR, March 2018. URL
mlr httpa//cc..aa..aaep.eeee/e/e///a html ISSN:
2640-3498.
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 19667-19679. Curran As- sociates, Inc., 2020. URL
nwurips cc/ httppppee/22//haaa//// html
#3b21256183cf
Jacob C Walker, Ali Razavi, and Aaron van den Oord. Predicting video with (vqvae), 2021. URL https://openrevier net htt.:/oooomm---DD11TBTTS
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9588-9597 Nashville, TN, USA, June 2021. IEEE. ISBN 978- 566444-00-444 doi: 10.1109/CVPR44447.000 2022. 01177.221.00000 000947. URL ieee org htt.://eeeoeeeeoccceeeee/////0e
Jason Yosinski, Jeff Clune, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In JCML Workshop on Deep Learning, 2015.
12

---

Under review as a conference paper at ICLR 2023
My Eg
q
p Decoder
L2
Input
Encoder
Gaussian sampling
Reconstruction loss
2,
KL
fi(ypac(o) + nu Vi
KL-divereence
Structural Equation Model (SEM)
Figure 8: Overview of the causal-prior VAE architecture.
APPENDIX
A PROOF OF DAG MAXIMALITY
Proposition 5. The set of all SEMs S = {(2 f((ppa (1) f3 with arbitrary DAGs G and probabilistic functions f contained within the set S {(2, f((=)....-))) [f], up to reorderings of the variables 2.
Proof. Because G is DAG it has an acyclic ordering of its vertices (Bang-Jensen & Gutin 2009);
;
i.e. if G has vertices V {V1, ..., J we can always construct sequence (Z1, 2n such that the set Z = {=1,... .... 2n } is mapped one-to-one to the set V. and such that 2; is an ancestor of 2j (i.c. there exists path from 2; to 2j in G) for all i <. j. Therefore. the set of parents (ag has to be subset of the set (zeli- < j}. from which Proposition follows.
B JOINT IDENTIFIABILITY OF REPRESENTATION AND CAUSAL MECHANISM
Before proving our identifiability result, we must first introduce a lemma about the commutativity of certain piece-wise linear functions.
Lemma 6. Consider a Leaky ReLU activation ierivaationfunction Me el al 2015), defined as:
[Cx i 2 0
Re,((x)
(12)
Dr ifx O0
with constants > 0,C 1, 0, 1, + D. Then the class of functions & that commute
with R, i.e.
= -cc.)))))c))) Yx
(13)
the set of monotonic piece-wise linear homogeneous functions with one piece in each half-plane, i.e.
Ax if 0
δ(x) =
Bx ifr < A.B
(14)
Proof. To solve the commutativity identity in Eq. {x 2 0, x < 0} (Ø(x) 2 0, 0z)) < <}:
13 we partition it into four domains, namely
x 0, ((x) <0 If < 0, ((x) 2 0 =
Cő(x) (Crx)
((x) Ax
(15)
Cő(x) = ((xx)
((x) = 0
(16)
x 0(x) <0
Dō(œ) o(Cx)
( 0
(17)
x 2 0, (,)()
=
DØ(x) = ((x))
((x) = Br
(18)
Equations 15 and 18 amount to solving the equality Ef(x) = of (Ex) which is satisfied for any
homogencous linear function ó assuming E * 1. Equations 16 and 17 amount to solving the equality
Ef(x) = ((Fx) which is only satisfied for ((x) = 0 assuming + 1, # F. Combining the results from Equations T5 18 the result follows.
13

---

Under review as a conference paper at ICLR 2023
We now introduce our main result.
Theorem For a causal VAE model (Eq. 3 and Eq. 5 with y = 2), assume the following conditions:
1. The encoder q(2/2)) and decoder p(z) 2) are Gaussians (Eq. 4 parameterized by deep networks, containing a Leaky ReLU (Eq. [2] as its last layer and first layer, respectively.
2. The outputs of the SEM ft are de-meaned and normalized, i.e. they are composed with batch normalization (BN) operators: 2; (N((())(()) where BN(u)= (iwl)/ (ffw))//Vrrlly /Varfu), or
they have a fixed constant scale.
Then the model is identifiable up the following indeterminacies, i.e. denoting the optimal parameters by 8" there is a different set of parameters such that pa_(x) = Pe(x) only under the following learnable transformations:
1. Simultaneous shifts by and orthogonal transformations R of oulput, i.e.: Py(2)) & Pgg (Rr b), 12(=)) R2 [pp=))
the the encoder's inpul and decoder's ss
2. Latent variable permutations, i.e. reorderings of
3. If the SEM mechanisms fü contain additional symmetries (e.g. f:= fs o S for some operator S), then there will be indeterminacy up to application of that operator.
Proof. The training objective is to find the model parameters of the generative distribution pe (4) such that it matches the observed empirical distribution p(z). The parameters are assumed to exist the function class of po (x) is sufficiently expressive, as is the case in werrppaaaetterize deep networks. We also assume that there exists a true model 6" which generates the data distribution pa- (x). Then the claim of non-identifiability that there exists at least one different parameterization 0 for which po(x) = pg- (zr), i.e. it is possible to learn another model that also fits the true data distribution but has different parameters O. We can express the lower bound on the data distribution using the evidence lower bound (ELBO, Eq. 3
In po (x) 2 Ez=p(a)) [ea(( [In p=(2)) - (=][(<]
(19)
with encoder ga(= (=(x), decoder pa (1] 2. and learnable prior p=(z) generated using the SEM (slightly overlouding to include all parameters of the model).
find another set of parameters 0 that fit the data perfectly: (x()) = pg- (z). In that case Eq. 19 becomes a strict equality, implying the following two conditions:
Under non- non-identifiability.
assume that
In po- (I) Ex-p(2) (eja) po (a|2)]
(20)
0 E- [KL[go [=|]a) (2)x)]lpo<=)]]
(21)
In words, the expectation of the log likelihood (first term of right-hand side of Eq. 19) becomes
the exact log evidence In pa- (z), while the posterior (2/22: fits the prior po(2) exactly (their KL- divergence is zero). Furthermore, for the optimal model 0* we know that
In pa- (x) Ez=p(==) [ gu- (z x)) [In po- (z]=)]]
(22)
Note the subtle difference from Eq. 20 is to use 0* instead of 0. We can thus combine the results for both models 6* (Equation 22 and 0 (Equation 20 as
Ex-p(a) (a In (x]
Exsp(a))
Egp- (z|z) [In po- (z|=)]]
(23)
We will write the left-hand side (LHS) of Eq. 23 in a form that makes its equivalence classes more obvious, using function composition. In order to do that, we first recall Eq. which we rewrite here (making explicit) for convenience:
(=(2)>) = (N(])o(()))) Ex(x))
(24)
The reparametrization trick, which is at the core of VAE implementations, allows to sample from the probabilistic encoder go(2[z) with a deterministic function (x,((x) (typically deep network) that takes samples from a fixed normal distribution and applies an affine transformation them:
(x.=(x) = (0(x)) + Eo(x) e, c~ 10.
(25)
The inner expectation in the LHS of Eq. 23 can then be written as:
Eg (=|z) [I po (<|=)) E (x))
(26)
14

---

Under review as a conference paper at ICLR 2023
By Eq. 4 the decoder pa pa(x)22 is also Gaussian with parameterized mean (2 so Eq. 23
equivalent to:
I Peppp( R (olaa In (x)(2) = Rewpl() (e
(
1-1lkb(=)) ==1²²
(27)
Bxxp(()) (z)o)))) (1)(2))) (=(x)===-) 1| x1|²
(28)
absorbing constants into 7. Finally, the LHS of Eq. 23 is then equivalent to the expectation of composition of deterministic functions (applied from right left):
Ex-p(t))e=pte)lee μg q8, o zr].
(29)
This makes explicit the order of operations in computing the reconstruction loss, and separates out all -on-deteriinistic elements into the expectation (namely by using the reparameterization trick with e < p(c) = (0, 1)).
We can now enumerate the equivalence classes of Eq. 23 by inserting identity operators g - h- o h (for arbitrary invertible functions h) between compositions of learnable functions in Eq.
29
Inserting these identities, Eq. 29 is equivalent to
Ee-p(a)) L (h o h h pg (g
g) qo. (ho h-oh-t) 2]
(30)
= ((L o h) (h-1 n og" (g qo h)
(31)
P
48.c
where grouped the arbitrary functions with the decoder as Fg. with the encoder go.e, and with the input as T. We can then apply the substitution F = h-1 o using the fact that
T (x') ||(x'') M()))l 2 Lher (h oh)
(32)
only true if h is orthogonal linear transformation plus constant (since Euclidean distances are invariant only under generalized rotations and translations) and thus obtaining the equivalent reparameterization
Ex=-(()) La o Pi qo. z).
(33)
Thus the function class of h must necessarily be restricted to orthogonal transformations, induced by the Euclidean structure of L (Eq. 32).
As for g. it is restricted by the fact that must be absorbed into the encoder, i.c. in Eq. 31 we must be able to group it with the previous encoder qa and define new encoder
To. o qa, o h
(34)
that can still be implemented as a deep network (of the same function class as go.)). Since we require that the encoder be followed by Leaky ReLU Ro,D (assumption of Theorem 7). for this to be true, g must commute with RC,D, i.c.:
d0.. go o q0 o h = g RoD o Ro. o goo hh
(35)
ge.c
where the first equality is taken from Eq. 31, in the second equality we decompose the network qo.c into its final Leaky ReLU Ro,D and the rest of the network % and finally in the last step we use
the commutativity of and Re.D D (Lemma. (). Since by Lemma. oonly monotonic piece-wise linear homogeneous functions commute with Rc this restricts the class of admissible functions for g to that class.
An identical conclusion follows for g1 as long as the first layer of the decoder is also Leaky ReLU (by assumption 1).
Assuming identity covariance for simplicity, common assumption in implementations, and which does materially change the result.
2This is the same reason why the function h and its inverse are used twice in Eq. 30 instead of inserting two different functions, e.g. hj and h2 and their inverses; the same generalized rotation h must be applied to both inputs of the Fuclidean distance Eq. 32
15

---

Under review as a conference paper at ICLR 2023
This means that the latent variables non-identifiable up to g implementing individual rescaling of each variable and variable permutation, unless we impose further constraints. We can impose the SEM to fix the rescaling, namely assuming that the output of each
mild structural constraints (oas() has fixed scale. This can be achieved by either: 1) batch-normalization; 2) SEM with no learnable scale. Batch normalizaaii,,,nn,, which is used by many common deep network models, fixes the distribution mean to zero and the variance to one for each dimension, removing scaling and shifting Segrees-of-freedoon As special case, the quadratic model used in Section [.I]] does not have learnable scale parameters. The only remaining degrees of freedom are variable permutations, which are unavoidable in latent variable models.
Having characterized the equivalence classes of Eq.
20 we must now consider those of Eq. 21 Using
the same reparameterization trick as in Eq. 25 the
KI. divergeee in Eq. 21 equivalent to
KL [po[2]] = E [g [In (o(()=))
In (o
(36)
(Eep(()) (o. - po (x))
(37)
Therefore, we can also express it using a composition of operators:
Ex-p(x))e [L q6,. L u qu, x]],
(38)
with nd the mean of the Gaussian output by f (by Theorem 2). assuming variance one as before for simplicity. We now insert the same identity operators and follow identical derivation Eq. 33] which recovers the exact same equivalence classes as before. The only identity that can be added differently from Eq. 30 will be an operator S for which 2i is invariant (11g 1i o S), so indeterminacy up to such symmetries of f the only other possibility.
C PROOF OF MEAN AND VARIANCE PROPAGATION FOR LINEAR ANMS
Theorem Consider a linear ANM defined as 2, = at 21....--1 + b; + nis with a, Ri-1, by E R and n4 N(ng]0, of) for i ...... d. Missing edges the causal graph can be represented as zeros in ag. Then the ANM's 's joint probability is given by p(z) = [2z Mr 2) with
2
2
2
=
II 1idd
A
b,
E
II b=
A
diag (o
II A
(39)
l=.....dd
v=dd
1(--1)((i1)
O111
((-
b1
4;
of
1
O
b=
((--0x((-11)
Oa-i
l(a-ix((d-))
ba
where Taxk denotes an identity matrix, while Ok and Okxt denote a zero column vector and a zero matrix, respectively.
Proof. We can write the ANM 2; a. 21....i-1 + by + n, in recursive multivariable form as
Zl..... 1 1(1-1)x(6-1)
O
0,-1
by
+
m4
(40)
Applying the formula for linear combination of Gaussians
Bishop
2006 Ch. 8.1.4)
& ~ (Z Ex). y N(yluy, E.y) - Ax+ Bytc ~ N(x Apz BBuy AE, AT BE. B7d)
(41)
to eq. 40 we can express the mean and covariance of 21.... as a function of the mean and covariance
for 2 -1 as
1
((i-1)x(i-1)
[001
0+-1
P1....
a
bi
(42)
1(i-1)x(i-1) di
(((-1)((E-1)
17
0
0
Σ E
E Enmni11
+
0
1
1
16

---

Under review as a conference paper at ICLR 2023
where we have used
Pl..... = E [3......] Ei..... = E8 (=1..... - (21.... [I.....)
(43)
and for the noise variables n holds that EE [n] = 0 and E [nn" = diag, (o,]). Now assume that eq.
39 holds for some d k. Inserting expressions for fi,..., and Σ, Ei,.... from eq. 39 into eq. 42 and extending the matrices with zeros and ones we obtain
Lack d
2
Ard==
Ok
bi....k 0
O
Ml..k+11
O
1
+
bx+1
(44)
i=k
Txcc
O
2
di
O
1)
diag
(o
O
E+....+11=
boonk Ok
ax+1
O
0
li=k
1)
T
II k=
A
O
O
a
Ok
1
+
O
O
o
Identifying
Teck T ak+1
Ok
Ar,d=
bi...,
As+1,d=h+1
A
0%
1
b1....++1
bx+1
(45)
eq. 44 becomes
M1...,k+1
II
Ard=kh1
b
(46)
\i=k-1
T
E1...
Ai,d=441
diag (of)
II
Ai,d=k11
(47)
vi+++
which completes the inductive step to obtain
k 1. Finally, we apply eq.
41 to relation 40 for d
1
0
11,2
P1
+
A2b1.
(48)
a2
T
T
E1,
1
Σ1
02
A o3)
02
02
where we have used μlr bi and Σ, of which shows that eq. 39 holds for = Therefore, by the induction principle the relation 39 holds for all d > 2.
D PROOF OF LINEARISATION OF NON-LINEAR SEMS
Theorem 9. The best linear approximation (in the beast-squares sense) of an ANM 2; fe(21...111) (eq. 5 with missing edges in the causal graph corresponding ignored in- puts in fe), around a pivot point 2a, is given by eq. 39 (Theorem 8 with
(f((21....111)
04 021,...
and b, Se(=f....-1) + aT -
(49)
where = = ((11...-1) + na and n. N(0, of) with learnable os.
Proof. By Taylor's theorem, expanding fe(E1....E-1) + n. around 26 2 up to first order gives
Je(2)...-11) 2e Ji(=i..... + + + (21,...-1 - 2 = Of (ff(211... (f(
(50)
of 2
b;
(51)
1-1
where a; and by are given by eq.
49 We can now use this lincarisation of fa to define a linearised SEM as of 21....i11 + b, and using this with Theorem the result follows.
Li
17

---

Under review as a conference paper at ICLR 2023
Reconstr. loss
Pru Zm
Zr1
Qm
Encoder (time-inv.)
Prame T
Hqu Zgq
My.1 Eqg1
L2
Frame T-1
4
Za
Encoder
Frame T-2
H.E
KL
zi f(( + n,,
Kl-divergence
Structural Equation Model (SEM)
Figure 9: The temporal causal-prior VAE architecture, for sequence of 3 video frames (plus random frame to represent time-invariant information, such as background).
0.996
0.995
0.994
0.983
6.992
0.991
10' confficient
10'
penallsation coeffidint
Figure 10: Reconstruction accuracy as a function of parameter controlling graph sparsity.
Figure 11: Structural Hamming Distance as a
function of parameter controlling graph sparsity.
E TRAINING DETAILS FOR THE CAUSAL-PRIOR VAE
We used custom dataset of 76800 binary images of size 64 64 containing ovals generated different scales s, 40 rotations r, 32 horizontal positions æ and 10 vertical positions y, where y x² + ny and and r factors sampled independently. We trained two isotropic-prior VAEs for comparison, one with a lower 3 and one with higher 3 (where / denotes the factor multiplying the KL divergence 3-layer MLP with 64 hidden ReLU units in each layer; the decoder is identical but with 4 layers, and there are latent variables z. We use the Adagrad optimizer (Duchi al. 201T) with learning rate 0.003 on batches of 100 samples, until convergence. The architecture is shown in Figure
causal-prior VAE and
defined in (Higgins et al., (016a))) The encoder is
F TRAINING DETAILS FOR THE TEMPORAL CAUSAL-PRIOR VAE
We created custom dataset of 3-frame 20 20 px video sequences of the main Super Mario Bros character moving linearly in random direction and with random speed on different backgrounds, where the character's positions are given by 1 4/, 22, 9/2 ~ M77 12), .3 2 - I1, 3/3 2y/2 - V/ where and are the horizontal and vertical position frame and t/ is the uniform distribution. We train our causal-prior VAE with 12 variables, 4 per time frame, and allow the SEM to learn arbitrary linear relationships between them. The architecture is shown in Figure 9 Each dataset sample consists of tuple consisting background and 3 consecutive frames where the character moves linearly. Each of the frames is then encoded separately into a 4-variate Gaussian distribution
18

---

Under review as a conference paper at ICLR 2023
and these concatenated to form 12-variate Gaussian, which is compared with its closest match in the SEM-sampled local Gaussian distribution. The distribution then sampled and split into samples per time frame and these are concatenated with the background latents and decoded back into 3 frames which are then compared using T22 loss with the 3 frames on the input. Additionally, frames passed through the SEM to predict their value at the next
the latents from the first time frame and this is decoded and compared using L2 loss with the third frame on the input. The architecture shown in Figure
G QUANTITATIVE EVALUATION OF LEARNED TEMPORAL GRAPHS
Reconstruction accuracy vs. graph complexity Figure I0 shows for the temporal experiments the reconstruction accuracy achieved with the model as a function of the parameter controlling graph sparsity (-1/ In(s) 9) in Equation 7 after training for 250 epochs. The plot shows that for weak edge penalisation the reconstruction accuracy is good (around 99.7%) while if the edge penalisation is too large the accuracy drops (to around 99.1 1%). This is because for weak edge penalisation the graph is relatively dense which allows the SEM to model the time-based causal relationships between the variables and when the penalisation is too big the graph becomes too sparse to be able to model these relationships. Somewhere around the value of the coefficient 10 the graph becomes sparse as possible while still keeping the reconstruction accuracy high, and this is the area from which we select the graph.
Structural Hamming Distance vs. graph complexity Figure I1 shows for the temporal experi- ments the Structural Hamming Distance between the learned and the ground truth graph function of the parameter controlling graph sparsity (-1/ In(s) in Equation after training for 250 epochs. The SHD is computed by counting how many edges need be inserted or removed to obtain the ground truth graph (up to permutation of variables within each time step). In the range where the coefficient is below approx. 10 the edge penalisation is too weak resulting many edges (thus high SHD) and in the region above approx. 10 2 the graph becomes too sparse with no edges (thus also resulting in high SHD). The region between 10 3 and 10 2 corresponds to the region where the learned graph has exactly the same structure the ground truth graph (and thus SHD is zero).
graph with too
19


Reviews:
[{"review_id": "xhgwz7NpDEE", "reviewer": "Reviewer_qqHC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with learning causal relationship between latent variables of a deep generative model, used to model high-dimensional complex data such as images. This is an important and difficult topic in causal inference. The paper starts with background about VAEs, SEMs and then proceeds to describing their method. The proposed method consists in linearizing the SEM given by a \"maximal DAG\", in which the topological ordering is fixed, and each latent variable is densely connected to each of the ones appearing before it in the ordering. For linear SEMs, the likelihood function is directly computable. Then, two experiments on image datasets are presented, in which certain variables have causal relationships (e.g., position of sprites in images), and one would like the DGMs to learn those relationships. ", "summary_of_the_review": "I have found this paper to be concerned with an important problem, and proposing an interesting method. However, I think that the paper neither has solid theoretical discussion of the setting or of the accuracy of the method, and also only has weak empirical evaluations. Therefore, I am leaning towards rejection. I am looking forward to the discussion period w/ authors, as well as reviewers. ", "strength_and_weaknesses": "Strength:\n\n(1) The problem that the authors are interested in is important. \n\n(2) The maximal dag idea allows to circumvent a lot of the cumbersome machinery in causal discovery learning for acyclicity, and search for topological ordering\n\n(3) The idea of linearization is interesting (I had not seen it in causal inference before)\n\n\nWeaknesses:\n\n(1) There is a discussion of identifiability in the introduction regarding the structural equation model for the latent variables (supposedly treating them as observed). Then, there is another discussion of identifiability in the setting of decoder + SEM setting, pointing out some previous work. However, to the best of my understanding, the 2008 paper is concerned with observed random variables. Because here the zs are latent, I do not think the theory actually holds. If it does, please explain it in details in the paper, as it is not straightforward. Also, the setting with temporality is another instance, and identifiability could hold in one setting, but not the other. \n\n(2) A discussion of time complexity is missing. In particular, what is the complexity of putting together the covariance matrix detailed in equation (8), and how does it affect scaling with respect to the number of latent variables? It may look prohibitively expensive for large dimensions. \n\n(3) There is no discussion of whether the algorithm has any form of theoretical guarantees after using a lower bound, and after linearization.\n\n(4) The experimental section seems rather like a discussion of disentanglement capabilities, rather than causal discovery learning. Why was 5 latent variables used in the first experiment, and not 2? After all, there are only one cause and one effect latent variable? Similarly, if the paper goes through all the process of explaining how to fit VAEs with latent SEMs, I would expect experiments with evaluation of the graph quality (e.g., does the method discover a sparse set of interactions between latent variables, is the conditional distribution exactly parabolic). \n\n(5) I could not find quantitative evaluation of methods in the main text. If the authors claim the VAE is identifiable, then could they report MCC and other metrics imported from non-linear ICA? I would also expect the authors to report metrics over the learned causal graphs for example.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tAJjlaTidUd", "reviewer": "Reviewer_mc4c", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of causal representation learning from observational data without any form of supervision. The authors assume that the latent causal structure follows a nonlinear additive noise model (ANM) and claim that it is then identifiable. In practice, they use a VAE with a causally structured prior. One novelty in this is that they propose to approximate the KL term in the ELBO by linearizing the causal mechanisms. Their approach is demonstrated on toy image data as well as time series data from a video game.", "summary_of_the_review": "If the author's claim is true, this would be a great result. Unfortunately, I do not think that the main point – that causal representations can be identified from observational data if we just assume the latent causal structure to be a nonlinear ANM – is correct. I hope the authors can point me to what I'm missing, add an identifiability argument, or provide strong empirical evidence for this claim. Otherwise, I do not think that this paper can be accepted.\n\nUPDATE after rebuttal and discussion: I am grateful to the authors for constructively and quickly engaging with my questions and criticisms and even working out a new identifiability claim. Unfortunately, I am convinced that this claim is incorrect, but even if it were true, it would not solve the main issue with the paper. I am afraid that this paper will need more substantial work before acceptance, and recommend rejection at ICLR.", "strength_and_weaknesses": "Strengths:\n- The problem of causal representation learning is interesting and potentially very impactful.\n- The data regime that the authors tackle – unsupervised observational data – is the holy grail of causal representation learning: a strong identifiability result here could be immediately practically relevant.\n- The VAE ansatz with a fully connected graph is very sensible.\n- The paper is generally well written. I appreciate the thorough introductions to the background material.\n\nWeaknesses:\n- Unfortunately, I believe that the main premise of the paper is flawed. (I hope I am wrong and the authors can correct me!) The authors correctly point out that when the causal variables are given, nonlinear ANMs are identifiable. They then claim that this implies that *causal representations with a latent nonlinear ANM are identifiable*, i.e. one can identify both the map from low-level data to causal variables and the causal graph from data. As far as I understand it, they do this without making assumptions on the decoder (like that being linear). I am afraid this claim of identifiability is not correct.\n  - One known counterexample is the special case of the trivial graph. Locatello et al [\"Challenging common assumptions in the unsupervised learning of disentangled representations\", ICML 2019] and some other papers show that in this case the variables are not identifiable.\n  - Another counterexample can be constructed from any nonlinear ANM with causal variables $z_i$ and noise variables $\\epsilon_i$. We can define a second nonlinear ANM that has a trivial graph and causal variables $z'_i = \\epsilon_i$ such that the observational data distribution remains the same. (Of course, the map to the data space will then be different between the two causal models.)\n  - If I'm missing something here, I kindly ask the authors to add an identifiability argument (for the representations, not just the causal structure once the representations are known) to the paper.\n- I don't quite understand the reasoning behind the linearizing approximation. Why not just use a nonlinear density estimator like a normalizing flow for each conditional probability density in the prior? Then we can approximate the KL quite well by sampling. Unlike the linearization, this approach should be unbiased (in some limit) and is well established.\n- I find it difficult to draw conclusions from the experiments. It would be better if the results were analyzed more quantitatively, showing that the learned representations and graphs are correct.\n- In the related work section, several recent works that use VAEs with causal structure in the latent space are missed, for instance:\n  - von Kugelgen et al, \"Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style\", NeurIPS 2021\n  - Brehmer et al, \"Weakly supervised causal representation learning\"\n  - Ahuja et al, \"Interventional Causal Representation Learning\"\n  - as a review, it may also be useful to point to Schölkopf et al, \"Towards Causal Representation Learning\", IEEE", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bxRkwBNeAB", "reviewer": "Reviewer_8bM1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a VAE-like framework with SEM to represent causal relationships among latent variables, and train the model by linearization.  Experiments on images and video demonstrate an ability to discover basic causal relationships. \n", "summary_of_the_review": "I enjoyed reading this paper, and found the exposition clear and results interesting.  I see it as a solid contribution.  \n\nQuestion: In Figure 4, why do z6 and z7 not depend on z2 and z3? \n", "strength_and_weaknesses": "Strengths: \n- Novel approach to modeling, building on well-established techniques like VAEs. \n- Experimental results support the authors' claims.\n- Well-written paper.\n\nWeaknesses: \n- would have been nice to see experiments on more complex data sets \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NW0ydJk63h", "reviewer": "Reviewer_WspD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In traditional causal inference, one often assumes access to the structured variables and the task is to discover causal relationships between them. In recent years, there has been a growing amount of interest to tackle the question when these variables themselves are unknown and this task is termed as \"causal representation learning\". To this end, the first task is to infer these ``causal variables\" themselves. Some existing works such as the causal VAE have tackled this question but in the context of linear structural equation models for the latents. In this work, the authors extend it to general non-linear SEMs. The authors leverage additive models as they can be identified with observational data. The method proposed by the authors extends standard VAEs that leverage independence of latents as prior to a non-linear SEM driven prior. To train VAEs, it is far more efficient to have a closed form expression for the KL divergences, which one cannot with standard non-linear SEM based prior. To tackle this the authors propose to break SEM into a piecewise linear SEM. The authors validate their method on synthetic datasets.", "summary_of_the_review": "The current work proposes a new class of VAEs that accommondate a SEM based prior. The authors propose a variational approximation for it. However, the method used for arriving at the approximation is not described, central parts of it namely pivot selection is missing. The main contribution of the paper is supposed to be non-linear additive SEMs but the main experiments are conducted with linear SEMs. Key comparisons with other works are missing as well. ", "strength_and_weaknesses": "**Strengths**\nThe authors study an important and timely problem. Currently, we lack methods that can recover latents with learnable causal priors.\nThe authors have done a good job of exposition starting with basics of the VAE and building the structure of the paper. \n\n**Weaknesses**\n\nI have many concerns with the paper that I highlight in a bulleted form below. \n\n1. **Method for local linearization** The authors state that they make a piecewise linear approximation of the non-linear SEM around pivot point. The authors do not state how they select the pivot points. Since the approximation is valid in a neighborhood only, it is important to have sufficiently many pivot points. This is central to the paper and it is appalling that it is missing. Further, if there was such a method used a natural question to ask would be if the method has some identification guarantees or it is completely ad-hoc. \n\n\n\n2. **Experiments have major issues** There are two sets of experiments that authors carry out. In the first experiment the authors use a quadratic relationship between the latents. \n\n   In the first set of experiments the authors use the known quadratic prior. If the prior is not learned then the experiment is a mere sanity check and does not provide any valuable insight. In fact, if the model is already aware that the relationship is quadratic, then the SEM effectively becomes linear by treating z^2 as the parent feature. If the SEM becomes linear, then doesn't that go against the whole point of the paper. \n\n   In the second set of experiments, the authors resort to using a linear SEM again. I am not sure why authors do that. If the point of the paper is to achieve identification for non-linear SEMs why use linear SEMs. Also, if it is linear and Gaussian would you not run into non-identification issues. \n\n2.  **Crucial comparisons are missing** The authors operate in time series settings to conduct their main experiments. For these experiments, why do they not compare against the work https://proceedings.mlr.press/v177/lachapelle22a/lachapelle22a.pdf. In the work referenced the authors learn a causal graph in time. See Section 4, Figure 3 of the paper. The authors rely on time-sparsity as the regularization to achieve identification. \n\n3.  **Lack of theoretical guarantees**  As highlighted above, the paper does not provide valuable experimental insights so I wanted to comment on theoretical guarantees. The paper also does not provide theoretical guarantees either. For instance, many of the works that authors cite do provide identification guarantees. The current theorems in the paper do not provide any new insights. \n\n4. I would recommend the authors to do a significant revamp of the experiments. Also, it should be made clear how you select pivots and manage the non-linear case. These changes will improve the paper. Finally, any insights from the theory even for two variable case would be valuable. \n\n5. The authors in Section 4.6 for some reason state the identification guarantees for non-linear SEM are empirically shown. There are many works and this https://arxiv.org/pdf/1205.2536.pdf is one example where identification has been shown in multivariate case. \n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject