Causal normalizing flows: from theory to practice
Authors: Adrián Javaloy
Pablo Sanchez Martin
Isabel Valera
Venue: NeurIPS 2023 oral (2023)
Keywords: causality
causal inference
normalizing flows
identifiability
interventions
counterfactuals
TL;DR: Armed with identifiability results, we demonstraste how to use normalizing flows to capture a causal model and perform causal inference with it.

Abstract: In this work, we deepen on the use of normalizing flows for causal reasoning. Specifically, we first leverage recent results on non-linear ICA to show that causal models are identifiable from observational data given a causal ordering, and thus can be recovered using autoregressive normalizing flows (NFs). Second, we analyze different design and learning choices for *causal normalizing flows* to capture the underlying causal data-generating process. Third, we describe how to implement the *do-operator* in causal NFs, and thus, how to answer interventional and counterfactual questions. Finally, in our experiments, we validate our design and training choices through a comprehensive ablation study; compare causal NFs to other approaches for approximating causal models; and empirically demonstrate that causal NFs can be used to address real-world problems—where the presence of mixed discrete-continuous data and partial knowledge on the causal graph is the norm. The code for this work can be found at https://github.com/psanch21/causal-flows.

Causal normalizing flows: from theory to practice
Adrián Javaloy 1,+ Pablo Sánchez-Martin 1.2 Isabel Valera 1,3
'Department Computer Science of Saarland University, Saarbrücken, Germany
2 Max Planck Institute for Intelligent Systems, Tübingen, Germany Max Planck Institute for Software Systems, Saarbrücken, Germany
Abstract
In this work, deepen the of normalizing flows for causal inference. Specifically, we first leverage recent results on non-linear ICA to show that causal models are identifiable from observational data given causal ordering, and thus can be recovered using autoregressive normalizing flows (NFs). Second, we analyse different design and learning choices for causal normalizing flows capture the underlying causal data Sata-generating process. Third, we describe how to implement the do-operator in causal NFs, and thus, how to answer interventional and counterfactual questions. Finally, in our experiments, we validate our design and training choices through comprehensive ablation study; compare causal NFs to other approaches for approximating causal models; and empirically demonstrate that causal NFs can be used to address real-world mrblems-whee mixed discrete- continuous data and partial knowledge on the causal graph the norm. The code for this work can be found at https: //g wigggttuub..e www.tttt..ccoo.eeeeeooeoootttttt http://aancchaalaaaaeaa causal-floss
Introduction
German Credit
Checking account
Deep learning is increasingly used for causal reas- oning, that is, for finding the underlying causal rela- tionships among the observed variables (causal dis- covery), and answering what-if questions (causal inference) from available data [28]. Our focus in this paper is to solve causal inference problems us- ing only observational data and (potentially partial) knowledge on the causal graph of the underlying structural causal model (SCM). This is exemplified in Fig. 1, where our proposed framework is able to estimate the (unobserved) causal effect of externally intervening on the sensitive attribute (red and yellow distributions), using solely observed data (blue dis- tribution) and partial information about the causal relationship between features.
9.8
aberroationl
CVA mal den
do(
0.6
Probubilitty
0.1
0.2
0.0
Nagaris
Aserge
High
Figure 1:
Observational and interventional distributions of the categorical variable check- ing account of the German Credit dataset [8], and their estimated values according to causal normalizing flow. Xş a binary vari- able representing the users' sex.
In this context, previous works have mostly relied on different deep neural networks (DNNs)-e.g) nor- malizing flows (NFs) [23, 26, 27], generative adversarial networks (GANs) [20, 39], variational autoencoders (VAEs) [15, 40], Gaussian processes (GPs) [15], or denoising diffusion probabilistic to iteratively estimate the conditional distribution of each observed variable given its causal parents, thus using an independent DNN per observed variable. Hence, to predict the effect of an intervention in the causal data- generating process, these approaches fix the value of the intervened variables when computing the new value for their children. However, they may also suffer
models (DDPMs) to]-to
"Conresonnencce ajavaloy@cs. ani-saarland.com
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

from error propagation which whicc worsens with long causal paths paths-and high number of parameters, which is addressed in practice with ad-hoc parameter amortization techniques [26. 27J. Moreover, several approaches also rely on implicit distributions [2, 20, 27, 32], and thus do not allow evaluating the learnt distribution.
In contrast, and similar to [16. 32, 34, 41], we here aim at learning the full causal- generrttnnnrtinngg process using single DNN and, in particular, using a causal normalizing flow. To this end, we first theoretically demonstrate that causal NFs are natural choice to approximate broad class of causal gata-gener processes ($3). Then, we design causal NFs that inherently satisfy the necessary conditions to capture the underlying causal dependencies (54), and introduce an implementation of the do-operator that allows us to efficiently solve causal inference tasks ($5). Importantly, our causal NF framework allows us deal with mixed continuous-discrete data and partial knowledge on the causal graph, which is key for real-world applications. Finally, we empirically validate our findings and show that causal NFs outperform competing methods also using a single DNN to approximate the causal data-generating process ($6).
Related work To the best of our knowledge, the closest works to ours are [16, 34, 41], as they all capture the whole causal data- gettrratngnntgg process using single DNN. Our approach generalizes the result from Khemakhem et al. [16], which also relies on autoregressive normalizing flows (ANF), but only considers affine ANFs and data with additive noise. In contrast, our work provides tighter connection between ANFs and SCMs (affine or not), more general identifiability results, and sound ways to both embed causal knowledge in the ANF, and to apply the do-operator. Another relevant line of works connect SCMs with GNNs [32, 4 41], and despite making little assumptions on the underlying the GNN are performed by severing
SCM, they lack identifiability guarantees, and interventions the graph, which we show in App. C may not work in general. Nevertheless, it is worth noting that the way we use A in the network design ($-4 is inspired by these works.
Preliminaries and background
2.1 Structural causal models, interventions, and counterfactuals
structural causal model (SCM) [28] is a tuple M = (ř, P.)) describing a data-generating process that transforms a set of d exogenous random variables, u ~ Pu. into a set of d (observed) endogenous random variables, x, according to f. Specifically, the endogenous variables are computed as follows:
11 = (u,, U2 u uz) Pu:
x; (xxxp... u:),
for i 1, 1, d.
In other words, each i-th component of f maps the i-th exogenous variable u, to the i-th endogenous variable x, given the subset of the endogenous variables that directly cause x;, Xpa, (causal parents).
An SCM also induces causal graph, powerful tool to reason about the causal dependencies of the system. Namely, the causal graph of an SCM M = (f, Pu) is the directed graph that describes the functional depend- encies of the causal mechanism. We can define the adjacency matrix of the causal graph A == fxf(x, u) * O, where O is the constant zero function, and the comparisons are made elementwise. Furthermore, the direct causes of the i-th variable (pa, in Eq. 1) are the parent nodes of the i-th node in A, and the ancestors of this node (which we denoted by an;) are its (in dirieet causes. See Fig. 2 for example of causal chain.
π ==
(1
2 0
3)
0
0 0
A
1
0
0
1
0
Figure 2: Causal graph, and its causal ordering ' and adjacency matrix A.
In the case that A is acyclic, we can pick causal ordering describing which variables do not cause others, and which ones may cause them. Namely, permutation x is said to be causal ordering of an SCM M if, for every x; that directly causes Xy, we have (1) < T()). Note that this definition equals that of a topological ordering and, without loss of generality, we will assume throughout this work that the variables are sorted according causal ordering.
Importantly, besides describing the (observational) data- gettraeenratiigg process, SCMs enable causal inference by allowing us to answer what-if if questions regarding: i) how the distribution over the observed variables would be if we force a fixed value on one of them (interventional queries); and ii) what would have happened to a specific observation, if one of its dimensions would have taken a different value (counterfactual queries).
2

---

Structural equivalence To reason about causal dependencies, we introduce the notion of structural equivalence. We say that two matrices and are structurally equivalent, denoted S R. if both matrices have zeroes exactly in the same positions, Similarly, we say that S is structurally sparser than R, denoted S 3 R. if whenever an element of R is zero, the same element of S is zero.
2.2 Autoregressive normalizing flows
Normalizing flows (NFs) [25] are a model family that express the probability density of set of observations using the change-of-varialee
rule. Given an observed random vector x of size d, normalizing flow a neural network with parameters 0 that takes x as imput, and outputs
(x) -~ Pa with log-density
log (og)(x) ITg (x)) + log/det( VxTo(x))), (2)
where Pu is a base distribution that is easy to evaluate and sample from. Since Eq. 2 provides the log- likelihood expression, naturally leads to the use of maximum likelihood estimation (MLE) [1] for learning the network parameters 0. While many approaches have been proposed in the literature [25], here we focus on autoregressive normalizing flows (ANFs) [18, 24). Specifically, in ANFs the i-th output of each layer of the network, denoted by z!z is computed as
7, hh)), where = d<
(3)
and where and a; are termed the transformer and the conditioner, respectively. The transformer is strictly monotonic function of 2 while the conditioner can be arbitrarily complex, yet it only takes the variables preceding 2; input. As result, ANFs have triangular Jacobian matrices, V VxTo(x).
Causal normalizing flows
Problem statement Assume that we have a sequence of i.i.d. observations X = {I1, 22 xN generated according to unknown SCM M. from which we have partial knowledge of its causal structure. Specifically, we know at least its causal ordering n, and at most the whole causal graph A. Our objective in this work is to design and learn an ANF Te, with parameters 0, that captures M by maximizing the observational likelihood (MLE), i.c.
N
maximize 0
log p (To (zx)) + log/det( Vv (x.)))
N
and that can successfully answer interventional and counterfactual queries during deployment, thus enabling causal inference. We refer to such model as a causal normalizing flow.
Assumptions We restrict the class of SCMs considered by making the following fairly common assumptions: i) diffeomorphic data- setiraagenraingggg process, i.e., f is invertible, and both and its inverse are differentiable; ii) no feedback loops, i.e., the induced causal graph is acyclic; and iii) causal sufficiency. i.e., the exogenous variables are mutually independent, p(u) = II. p(u.).
SCMs as TMI maps To achieve our objective, and bridge the gap between SCMs and normalizing flows, we resort to triangular monotonic increasing (TMI) maps, which autoregressive functions whose i-th component is strictly monotonic increasing with respect to its i-th imput. TMI maps hold a number of useful properties, such as being closed under composition and inversions. Conveniently, layer of an ANF (Eq. 3) is a parametric TMI map that can approximate any other TMI map arbitrarily well, which makes ANFs also TMI maps approximators:" a fact that has been exploited in the past to prove that ANF' are universal density approximators [25].
We now show that any SCM can be rewritten TMI maps, and P. is the set of all fully factoiiat distributions, p(u) = II, p(u,). First, given an acyclic SCM M (f, Pu) with f: X x U - X as in Fqq. 1, we can always unroll by recursively replacing each x; in the causal equation by its function f. (see Fig. 4b for example), obtaining an equivalent non-recursive function f U + X. This function f writes each x, function of its exogenous ancestors Uan; and, since M is acyelic, f is triangular map. For simplicity, assume that Pu is a standard uniform distribution. Then, following the causal ordering, we can apply a Darmois
tuple (f, P.) P.) Fx Pu, where F is the set of all
While it is common shuffie the inputs for each layer, keep the same order across the network.
3

---

construction [7. 13] and replace each function f. by the conditional quantile function of the variable x, given Xps, (which depends on uan,) ) eventually arriving to TMI map f. This procedure follows the proof for non- non-identifiabiiityy in ICA [13]. but restricted to one ordering. The case for a general Pa follows similar construction, but using a Rosee-l-t (KR) transport [19, 31] instead.
Isolating the exogenous variables Now that we have SCMs and causal NFs under the same family class -.e., the family F x Pu of TMI maps with fully-factorized distributions- we leverage existing results on identifiability to show that we can find a causal NF To such that the i-th component of (x) is a function of the true exogenous variable u, that generated the observed data. More precisely, note that, since we can rewrite M as an element of the family F x Pu. identifying the true exogenous variables of an SCM M is equivalent to solving non-linear ICA problem with TMI generators, for which Xi and -eddm [38] proved the following (re- stated to match our setting):
Theorem 1 (Identifiability). If two elements of the family F x Pu (as defined above) produce the same observational distribution, then the two data data-gennrating processes differ by an invertible, component- wise transformation of the variables
Thm. I implies that, if we can find causal NF (To, Po) E E Pu that matches the observational distribution generated by M = (f, PM) E F x Pu, then we know that the exogenous variables of the flow dif- fer from the real ones by a function of each component independ- h(u) =
Px
To
ently, i.e., (o((( ~ Po with u PM and To(f(u)) (hr (u,). hu).h2(us (2(())).... ha(ua)), where each h2 is an invertible function. Fig. 3 graphically illustrates Thm. 1. Furthermore, Thm. I also implies that the functional dependencies of the causal NF must agree with that of the SCM, i.e., that To needs to be causally consistent with M. We formally present this result in the following corollary (proof can be found in App. A), where I denotes the identity matrix:
PM
Po
h
Figure 3: Thm. as commutative diagram
Corollary 2 (Causal consistency). a causal NF To isolates the exogenous variables of an SCM
M. then V vxTo(x)= = I A and V T = I + > 2==1 matrix of M. In other words, To is causally consistent with the true data-generating process, M.
diam(A) A", where A is the causal adjacency
sketch of the proof goes as follows: since Fig. is a commutative diagram, we can write the result of To and Tg in terms of the true f. h. and their inverses. Then, we can use the chain rule to compute their Jacobian matrices, and since h has a diagonal Jacobian matrix, it preserves the structure the Jacobian matrices of f and its inverse. To sum up, we have shown that causal NFs are a natural choice to estimate an unknown SCM by showing that: i) both SCMs and causal NFs fall within the same family F x Pui ii) any two elements of this family with identical observational distributions are causally consistent; and iii they differ by an invertible component-we -ripieett-wieeeeee transformation.
3.1 Causal NF' for real-world problems
To bring theory closer to practice, we need to extend causal NFs to handle mixed discrete-continuove data and partial knowledge on the causal graph, which are common properties of real-world problems. Due to space limitations, we provide here a brief explanation, and formalize these ideas in App. A.2.
Discrete data To extend our results to also account for discrete data, we take advantage of the general model considered by Xi and Bloem-Reddy [38] that includes observational noise (independent of the exogenous variables), and consider a continuous version of the observed discrete variables by adding to them independent noise e E [0, 1] (e.g., from a standard uniform), such that the real distribution is still recoverable. Intuitively, our approach assumes that discrete variables correspond to the integer part of (noisy) continuous variables generated according to an SCM fulfilling our assumptions, such that both our theoretical and practical insights still apply.
Partial knowledge While we rarely know the entire causal graph A, we often have good grasp on causal relationships between a subset of observed variables-e. e.g., sex and age are not causally related-whin missing the rest. When only partial knowledge on the graph is available-i.e. we only know the causal relationship between a subset the observed variables, we can instead work with a modified acyclic graph A obtained by finding the strongly connected components as in [35], where subsets of variables with unknown causal relationships are treated as a block (see $7 for an example).

---

= Gx Ju
G3(G2(G,))
x= +G++++J
u= (I G)x
X
u
x
x
(a) Recursive.
(b) Unrolled.
(c) Compacted.
(d) Inverted.
Figure 4: Example of the linear SCM (xy== x2== + + written (a) in its usual recursive formulation; (b) without recursions, with each step made explicit: (c) without recursions, as a single function; and (d) writing as function of x. The red dashed arrows show the influence of b1 on X3 for all equations from u to x, with the compacted version exhibiting shortcuts (see $4). Note that in the linear we have A G o 0, and that G1, G2, G3 G + I are any three matrices such that their product equals G + G + I.
This allows us to reuse our theoretical results for known parts of the graph, thus generalizing the block identifiability results from von Kügelgen et al. [36].
4 Effective design of causal normalizing flows
We showed in $3 that causal NFs are a natural choice to learn the underlying SCM generating the data. Importantly. Thm. 1 assumes that we can find a causal NF whose observational distribution perfectly matches the true data distribution (according to the underlying SCM). In practice, however, reaching the optimal parameters may be tricky as: i) we only have access to finite amount of training data; and ii) the optimization process for causal NFs (like for any neural network) may converge to local optima. In this section, we analyse different design choices for causal NFs to guide the optimization towards solutions that do not only provide an accurate fit of the observational distribution, but allow us to also accurately answer to interventional and counterfactual queries.
Let us start with an illustrative example. Suppose that we are given the linear SCM in Fig. 4a, and we want to write the SCM equations as TMI map to approximate them with a causal NF. As discussed in $3, we can unroll the causal equations (Fig. 4b)-resulting a composition of functions structurally as sparse as I + A. These functions can be compacted into a single transformation (Fig. 4c), such that each x; depends on its ancestors, Uan: However, note that in this step shortcuts appear, making direct and indirect causal paths in this representation indistinguishable- our example, the indirect causal path from μ to X3 present in Fig. 4a and Fig. 4b does not go anymore through the path that generates X2, but instead via a shortcut that directly connects u to X3. Alternatively, we can invert the equations write as a function of (Fig. 4d), which is structurally equivalent to I A.
We remark that the above steps can be applied to any considered acyclic SCM (refer to App. B for more detailed discussion). In particular, and we can similarly reason about the causal dependencies through the Jacobian matrices of the generators, (x) and VuTo (u). Moreover, note that the diffcomorphic assumption implics that we can invert the causal equations. Next, inspired by the different representations of an SCM (cxemplified in Fig. 4), we consider the following design choices for causal NFs:
unroll the equations in finite number of steps,
Generative model The first architecture imitates the unrolled equations (Fig. 4b), i.c., the causal NF is defined as function from u to x. Importantly, when full knowledge of the causal graph A assumed, we also replicate the structural sparsity per layer by adequately masking the flow with I + A. In this way, the information from u to x is restricted to flow as if we were unrolling the causal model [34] and, as a result, the output of the I 1-th layer of the causal NF is given by:"
/ / T. (2):
where
h c,(2/a,))
Note that, by restricting each layer such that V (z) + A, there cannot exist shortcuts the optima. For example, in Fig. 4, the (indirect) information of uj to generate X3 by first generating
The order has been reversed w.r.t the causal NF definition from Eq. 3.
5

---

X2 needs to go through the middle nodes in Fig. 4b. However, as shown by [34], we need at least diam (A) layers our causal NF avoid shortcuts and thus differentiate between the different direct and indirect causal paths connecting a pair of observed variables.
et al.
In contrast, if we only know the causal ordering, then the causal NF will need to rule out the spurious correlations by learning during training the necessary zeroes to fulfil causal consistency (Cor. 2), i.e. diam(() A".
such that (x) = I + A and V T = I + > n=1
Abductive model Reminiscent to the abduction step [30], another natural choice is to model the inverse equations of the SCM as in Fig. 4d, hence building a causal NF from x to u. Under known causal graph, we can again use extra masking in the causal NF to force each layer I to be structurally equivalent to I A, such that
2 = T, (z,
where h h/ cz=c((Z)p)))
(6)
Remarkably, this architecture is capable of capturing all indirect dependencies u on x, even with a single layer. This is result of the autoregressive nature of the ANFs used here to build causal NFs, as they compute the inverse sequentially. In the example of Fig. 4, the indirect influence of i on X3 via X2 has to necessarily generate X2 first (Fig. 4a). Similar the previous architecture, in the absence of a causal graph (i.e., when only the causal ordering is known), the causal NF will need to rely on optimization to discard all spurious correlations.
4.1 Necessary conditions
We next analyse the necessary conditions for the design of a causal NF be able accurately approximate and manipulate an SCM. A summary of the analysis can be found in Tab 1.
Expressiveness The least restrictive condition is that the causal NF should be able to reach the optima and, as mentioned in $3, a single ANF layer (Eq. 3) a universal TMI approximator [25].
Identifiability In order to perform interventions as we describe later in $5, we need the causal NF to isolate the exogenous variables, so that we can associate them with their respective endogenous variables. As we saw in $3, if the causal NF is expressive enough, and if it follows a valid causal ordering w.t.t. the true causal graph A, then Thm. ensures that we can isolate the exogenous variables up to elementwise transformations.
Causal consistency As stated in Cor. 2, the causal NF needs to share the causal dependencies of the SCM at the optima, meaning that their Jacobian matrices need to be structurally equivalent, i.e., (x) =
A (Fig. 4d), and VuTo (u) Em=r diaam(A) (A)
A" Th + I (Fig. 4c). Given the (partial) Idlam(() A + I
causal graph A, the generative model in Eq. 5 by design holds (u) for any sufficient number of layers I. Z diam (see [34, Prop. 1J), however, there might still exist spurious paths from x to u. Similarly, the abductive model in Eq. 6, while may not remove all spurious paths from x to u if L > 1, ensures causal consistency when L
In cases where the selected architecture for the causal NF does not ensure causal consistency by design, but we have access to the causal graph, we can use this extra information to regularize our MLE problem as
minimize
Ex |- log + (o @ (1 A)ll2]
where 1 is a matrix ones, thus penalizing spurious correlations from x to Tx(x).
Tab summarizes the discussed properties of the considered design choices. Remarkably, the abductive model with a single layer (similar to Fig. 4d) enjoys all the necessary properties of a causal NF by design. That is, the abductive model with the provided causal graph A, greatly simplifying the optimization process.
is expressive, and causally consistent w.r.t.
Remark. It is not straightforward why abductive models can be causally consistent by design, but generative models cannot. The answer lies on the structure of the problem. Intuitively, this is a consequence of the mapping x u being structurally sparser (u, depends pa,, see Fig. 4d) than that of u - x (x, depends on an,, see Fig. 4c). Therefore, ensuring causal consistency from u to x does not necessarily imply causal consistency from x to

---

Table Summary of the considered design choices, their induced properties, and their time complex- ity for density evaluation and sampling. Generative models design their forward pass as u + X, and abductive models as x - u. See $4 for an in-depth discussion.
Design Choices
Model Properties
Time Complexity
Causal Asumption
Causal Consistency
Network Type
Sampling
Evaluation
1
x
X
Generative
Ordering
X
O(L)
O(dL)
u
Generative
Graph A
/
X
O(L)
(ddL)
Abductive
Ordering
X
X
OdL))
O(1)
u
Abductive (L>1) 1)
Graph
X
X
QdL
O(L)
Abductive =1
Graph /
V
((dL)
O(L)
5 Do-operator: enabling interventions and counterfactuals
In this section, we propose an implementation of the do-operator well-suited for causal NFs, such that we can evaluate the effect of interventions and counterfactuals [28]. The do-operator [30]. denoted as do(x; a). is mathematical operator that simulates a physical intervention on an SCM M. inducing an alternative model M that fixes the observational value x; = a, and thus removes any causal dependency X,. Usually, the do-operator is implemented by yielding SCM M (fz, Pu) result of replacing the i-th component of f with constant function, f Unfortunately, this implementation of the do-operator only works for the recursive representation of the SCM (Fig. 4a), thus not generalizing to the different architecture designs of causal NFs discussed in $4 the previous section.
We instead propose to manipulate the SCM by modifying the exogenous distribution P.. while keeping the causal equations f untouched. Specifically, an intervention do(x; a) updates Pa. restricting the set of plausible u to those that yield the intervened value a. We define the intervened SCM as . = (f, Pi)), where the density of Pf is of the form
(=) = = (( u:) (x) =a)) II,), JØC
(8)
and where 8 is the Dirac delta located at the unique value of u, that yields x; a after applying the causal mechanism f.. This approach resembles the one proposed for soft interventions [10] and backtracking counterfactuals [37]. Moreover, as shown in App. C, can be generalized for non- bijective causal equations. Note also that Eq. 8 well-defined only if the set {u ~ P. f1 u) a} non-empty, i.e., if the intervened variable takes plausible value (i.e., with positive density in the original causal model). Remarkably, this implementation works directly on the distribution of the exogenous variables and can be applied to any SCM representation (see Fig. 4), and therefore any of the architectures for the causal NF in $4.
Implementation details We take advantage of the autoregressive nature of causal NFs, and gen- crate samples from Eq. 8 by: obtaining the exogenous variables observational value by its intervened value, x; = at and iii) by setting to the value of the i-th component of (x). If the causal NF has successfully isolated the exogenous variables ($3), and it preserves the true causal paths ($4), then the causal NF ensures that x, = a independently of the value of its ancestors, since u; can be seen in Eq. 8 as deterministic function of the given a and the value of its parents (and therefore its ancestors). We provide further details and the step-by-step algorithms compute interventions and counterfactuals with causal NFs in App. C.
(x);; ii) replacing the
6 Empirical evaluation
In this section, we empirically validate the insights from $4, and compare causal NFs with previous works. Additional results and in-depth descriptions can be found in App. D.
7

---

Direction:
Model:
Ordering
Ordering
Graph
Graph"
1.00
Forward
RMSE ATE:
2.0
0.30
0.75
9.20
1.00
0.20
0.15
0.10
forwor
RAISE ATE
0.05
0.05
0.15
0.00
0.0
RMSE ATE
RMSE CF
(a) Ablation study on a 4-node chain SCM, as we vary the number layers L. Top: generative models (u + x). Bottom: abductive models (x + u).
(b) Comparison the two best models of cach row
Figure 5: Ablation of different choices of the causal NF to be causally consistent, and capture the observational and interventional distributions. The use of regularization on the Jacobian (Eq. 7) is indicated with the superscript. The abductive causal NF with information A and I. outperforms the rest of models across all metrics, demonstrating its efficacy and simplicity.
6.1 Ablation study
Experimental setup We evaluate every network combination described Tab on a 4-chain SCM (which has diameter 3 and very sparse Jacobian) and assess the extent to which these models: i) capture the observational distribution, using KT. po): ii) remain causally consistent w.r.t. the original SCM, measured via [(V= == [xx) (1 A)ll2 from Eq. 7: and iii) perform interventional tasks, such as estimating the Average Treatment Effect (ATE) [29] and computing counterfactuals, both of which we measure with the RMSE w.r.t. the original SCM. Every experiment repeated 5 times, and every causal NF uses Mask Autoregressive Flows (MAFs) [24] as layers.
Results Fig. shows the result for different design choices of causal NFs. Specifically, show: network design (generative u + x vs. abductive x + u), causal knowledge (ordering vs. graph), number of layers L, and whether to use MLE with regularization (Eq. 4 vs. 7).
First, we see in Fig. 5a (top) that, as expected, the generative models (u - x) using the causal graph cannot capture the SCM with 7, < diam A = 3. Furthermore, we observe that abductive models x - (Fig. 5a, bottom) accurately fit the observational distribution, and that embedding the causal graph in the architecture significantly improves the ATE estimation.
Second, we now compare the two network designs in Fig. 5b, and observe that in general abductive models results in more accurate estimates of the observational distribution, as well as of interventional and counterfactual queries. Finally, we observe that regularization works well in all cases, yet renders useless for the abductive model with L and knowledge on the graph, since it is causally consistent by design. In summary, our experiments confirm that, despite its simplicity, the causal abductive model with L. 1 outperforms the rest of design choices. As a consequence, in the following sections we will stick this particular design choice, and refer to it as causal NF.
6.2 Non-linear SCMs
Experimental setup We compare our causal NF (causal, abductive, and with L = 1) with two relevant works: i) CAREFL [16], an abductive NF with knowledge on the causal ordering and affine layers; and íi) VACA [34], variational auto-encoding GNN with knowledge fair comparison, every model uses the same budget for hyperparameter tuning, our causal NF uses affine layers, and CAREFL has been modified to use the proposed do-operator from $5 (as the original implementation only works in root nodes). We increase the complexity
the graph. For
the SCMs and
consider: i) TRIANGLE, a 3-node SCM with dense causal graph; ii) LARGEBD [11], 9-node SCM with non-Gaussiin P. and made out of two chains with common initial and final nodes; and iii) SIMPSON [11], a 4-node SCM simulating a Simpson's 's paradox [33], where the relation between two variables changes the SCM is not properly approximated.
The results are summarized in Tab 2. In nutshell: the proposed causal NF outperforms both CAREFL and VACA in terms of performance and computational efficiency. VACA shows poor performance, and is considerably slower due to the complexity of GNNs. Our causal NF outperforms
Results
8

---

Table 2: Comparison, on three non-linear SCMs, of the proposed causal NF, VACA [34], and CAREFL [16] with the oo-orerator proposed in $5. Results averaged over five runs.
Performance
Time Evaluation (us)
Dataset
Model
KI
ATERMSE
CFRMSE
Training
Evaluation
Sampling
TRIANGLE
Causal NF
0.00p.
0.120.00 os
0.130
0.520.or
0.58p or
1.070.12
CAREFL
0.00g.00
0.12a.on
0.17p.0s
0.57a.18
0,83o.20
1.68p.ez
[34]
VACA
p.70
4.780.01
4.19p.04
28.821.21
23.00p.55
70.653.70
LARGEBD
Causal NF
1.51o.04
0.02o.00
0.01o.00 .0
0.52 10
0.60,. 17
3.05p.e0
NLIN
CAREFL
1.51p.05
0.05p.01
0.08p.01
0.84p.47
1.18g. 17
8.25
[11]
VACA
53.662.07
0.39p.00
0.82p.u2
164.9211.10
137.8815.72
167 .42.
SIMPSON SYMPROD
Causal NF
0.00p.o0
0.07p.01
0.120.02
0.
0.60p.11
1.51p.30
CAREFL
0.00
0. 0.10p.02
0.1 0.17p.04
0.49
0.
1.9
[1]]
VACA
13.85p.64
0.896.00
1.500.04
49. 19.26
37 37.78a.41
79.2024.60
Table 3: Accuracy, Fl-score, and counterfactual unfairness of the audited classifiers. Causal NFs enable both fair classifiers and accurate unfairness metrics. Results are averaged on five runs.
Logistic classifier
SVM classifier
full
unaware
fair x
fair u
full
unaware
fair x
fair u
72.28
72.37
59 59.668.37
73. 73.08
76. 042.86
76. 76.80g
68 08.288.74
77.39
accuracy
67.00g.83
66. 666.75266
51.756.91
66.50s.70
69.50a.11
71.00a.sa
59. 59.252.99
69. 75..6..2
unfairness
5.842.93
2.81
0.00p.00
0.00p.00
6.652.45
2.78 2.780.40
0.00p.00
0.00p.00
CAREFL in counterfactual estimation tasks with identical observational fitting, showing once more the importance of being causally consistent. Even more, our causal NF is also quicker than CAREFL, as best-performing CAREFL architectures have in general more than one layer.
Use-case: fairness auditing and classification
To show the potential practical impact of our work, we follow the fairness use-case of Sánchez-Martin al. [34] on the German Credit dataset [8]-a dataset from the UCI repository where the likelihood of individuals repaying a loan is predicted based on a small set of features, including sensitive attributes such as their sex. Extra details and results appear in App. E.
Experimental setup As proposed by Chiappa [3], we use a partial graph which groups the 7 discrete features of the dataset in 4 different blocks with known causal relationships, putting in practice the results from $3.1. For the causal NF, we use the abductive model with a single non-affine neural spline layer [9]. Our ultimate goal is to train causal NF that captures well the underlying SCM, and use it to train and evaluate classifiers that predict the (additional) binary feature credit risk, while remaining counterfactually fair w.r.t. the binary variable sex, xs.
In this setting, we call a binary classifier & X + (0,1) counterfactually fair [21] if, for all possible factual values xf E X. the counterfactual unfairness remains zero. That is, if we have that Ex [P(n(x=f= 1 do(xs 1),x*) - (f do((x = x), = 0, where xe is counterfactual sample coming from the distribution P(xx do((x = s), x²), for 0,
Following Sánchez-Mart Martin et al. [34], we audit: model that takes all observed variables (fulf); an unaware model that leaves the sensitive attribute Xş out; a fair model that only considers non- descendant variables of xş (fair x); and, demonstrate the ability to learn counterfactually fair classifier, we include a classifier that takes u = (x(x) as input, but leaves uy out (fair u).
Results Tab 3 summarizes the performance and unfairness of the classifiers, using logistic regres- sion [6] and SVMs [S]. Here, we observe that by taking the non-sensitive exogenous variables from the causal NF, the obtained classifiers achieve comparable or better accuracy than the rest of the classifiers, while at the same time being counterfactually fair. Moreover, the estimations of unfairness obtained with the causal NF match expectations [21]. with full being the most unfair, followed by aware and the two fair models. With this use-case, we demonstrate that Causal NFs may indeed be a valuable asset for real-world causal inference problems.

---

8 Concluding remarks
In this work, we have shown-bow both theoretically and empirically- ttahtt choice to learn broad class of causal data- generratnn processes in principled way. Specifically, we have proven that causal NFs can match the observational distribution of an underlying SCM, and that in doing so the ANF needs to be causally consistent. However, as limited data availability and local optima may hamper reaching these solutions in practice, we have explored different network designs, exploiting the available knowledge on the causal graph. Moreover, we have provided causal NFs with a to-operator to efficiently solve causal inference tasks. Finally, we have empirically validated our findings, and demonstrated that our causal NF framework: i) outperforms competing methods; and ii) can deal with mixed data and partial knowledge on the causal graph.
-that causal NFs are natural
Practical limitations Despite considering a broad class of SCMs, we have made several assump- tions that, while being standard, may not hold in some application scenarios. With regard to our causal assumptions, the presence of unmeasured hidden confounders may break causal sufficiency assumption; mismatches between the true causal graph (e.g.. it may contain cycles) and our assumed graph/ordering may lead poor estimates of interventional and counterfactual queries; and the non-bijective true causal dependencies may invalidate our theoretical and thus practical findings. Besides, we have focused on MLE estimation for learning the causal NF. However, MLE does not test the independency of the exogenous variables during training, which would also break our causal sufficiency assumption.
Future work We firmly believe that our work opens a number of interesting directions explore. Naturally, we would like to address current limitations by, e.g., using interventional data to address the existence of hidden confounders [14, 23], explore alternative losses other than MLE (c.g.. flow matching [22]). Moreover, it would be exciting to see causal NFs applied to other problems such as causal discovery [11], fair decision-making [21], or neuroimaging [12], among others. However, we would like to stress that, in the above contexts, it would be essential to validate the suitability of our framework (e.g., using experimental data) prevent potential harms.
Acknowledgements
We would like to thank Batuhan Koyuneu and Jonas Klesen for their invaluable feedback, as well as the anonymous reviewers whose feedback helped us improve this manuscript. This project is funded by DFG grant 389792660 part of TRR 248 CPEC. Pablo Sánchez Martín thanks the German Research Foundation through the Cluster of Excellence "Machine Learning New Perspectives for Science", EXC 2064/1, project number 390727645 for generous funding support. The authors also thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Pablo Sánchez Martín.
Bibliography
[1] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006. (page 3)
[2] Patrick Chao, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan. Interventional and coun- terfactual inference with diffusion models. ArXiv preprint, abs/2302.00860, 2023. URL
sttpp://arxiv.org/abss 233.2 2302.
/0tp//00002.0...../ 00860. (page 2, 29)
[3] Silvia Chiappa. Path-specific counterfactual faireess. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firs Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in
Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, doi: 10. 10.1609/aaai. htt9:/aai.v33301... 333001..3011881. URL https:
pages 7801-7808. AAAI Press, 2019.
(page 9)
[4] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus). In Proc. of ICLR, 2016. URL http:://tttii..aaaaa
abs/1511.07288. (page 27)
[5] Corinna Cortes and Vladimir Vapnik. Support- networks. Machine learning, 20(3): 273-297, 1995. (page 9)
10

---

[6] David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series (Methodological), 20(2):215 232,, 1958. (page 9)
[7] G. Darmois. Analyse générale des liaisons stochastiques: etude particulière de I' l'analyse factorielle linéaire. Revue de l'Institu International de Statistique / Review of the International Statistical Institute, 21(1/)):2-8, 1953. ISSN 03731138.
URL http: http::/xww.jstor/ org/
stable/1405511. (page 4)
[8] Dheeru Dua and Casey Graff. UCI machine learning repository, 2021. URL https:// wwxiv.org/abs//220 http://233.0444 04786. (page 9, 29, 31)
[9] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7509-7520, 2019, URL https: hea2///ee71d33522004400077444222 Abtseaataa..saaaaaaatttt..t..t (page 9, 28, 29)
[10] Frederick Eberhardt and Richard Scheines. Interventions and causal inference. Philosophy of Science, 74(5)9981-995, 2007. ISSN 00318248, 1539767X. URL httpp:/wwwwjsttrr.orrg .rrg/
stable/100.086652563. (page 7)
[11] Tomas Geffner, Javier Antorán, Adam Foster, Wenbo Gong, Chao Ma, Emre Kıcıman, Ajay Sharma, A. Lamb, Martin Kukla, Nick Pawlowski, Miltiadis Allamanis, and Cheng Zhang.
Deep end-to-end causal inference. ArXiv preprint, abs/2202.02195, 2022. URL https:
/arxivv.rrg/ass//2222 2ttp//2222..2.....2
(page 8, 9, 10, 27,
[12] Krzysztof Gorgolewski, Christopher Burns, Cindee Madison, Dav Clark, Yaroslav Halchenko, Michael Waskom, and Satrajit Ghosh. Nipype: A flexible, lightweight and extensible neuroima- ging data processing framework in python. Frontiers in Neuroinformatiis, 5, 2011. ISSN 1662- 5196. doi: 10.3389/fninf 1... 1.0013. URL btttps://www.ffootiierree wwwp:rrssiiiirrrrrirrlllleeeeeee 10. 10.3389/fminf 0011.00013. (page 10)
[13] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429-453 -399, 1999. ISSN 0893-6080. doi: http://ooogg//o001111o/8/9-2...0 cccaacc//rrricee//////ss60000000
URL http:::://w..cccnnddrreetttt (page 4)
[14] Maximilian llse, Patrick Forré, Max Welling, and Joris M Mooij. Combining interventional
and observational data using causal reductions. ArXiv preprint, abs/2103.04786, 2021. URL
https://arxiv.or www.vv..r///aeo aabs/2103 037860...0... (page 10)
[15] Amir-Hossein Karimi, Bodo Julius von Kügelgen, Bernhard Schölkopf. and Isabel Valera. Algorithmic recourse under imperfect causal knowledge: a probabilistic ap- proach. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12. 2020, virtual, 2020. URL https: httttp:://proceeding..eeeee wwwparips.ccppppe//////// wtt.:eabbbboobbttttt....tttt (page 1)
[16] Ilyes Khemakhem, Ricardo Pio Monti, Robert Leech, and Aapo Hyvärinen. Causal autore- gressive flows. In The 24th International Conference Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learn- ing Research, pages 3520-3528. PMLR, 2021. URL http://proceedingg v130/ http///aaaaaeeemaaaamm.l.a (page 2, 8, 9, 23, 28, 29)
preas//
[17] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proc. of wtt..o/ggg/ggg/.......1111.e /iwp..111......... (page 24, 29)
ICLR, 2015. URL
[18] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: htsttpp://prooceedng..eeeete ddeebdeefdb7 (beeddeeftbrrrrrraasSeeeccaaadd Pdff (page 3)
[19] Herbert Knothe. Contributions to the theory of convex bodies. Michigan Mathematical Journal, 4:39-52, 1957. (page 4, 18)
[20] Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. Causal- gan: Learning causal implicit generative models with adversarial training. In Proc. of ICLR.

---

OvenReview.net, 2018. URL https:///ppaaeerssssaacccooooooa
(page 1,
[21] Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. Counterfactual fair- ness. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4066-4076, 2017. URL httplp::s//pooceedngss.eeeee wttpppp...cpppeeeeeeeeeeee ppppp///17//aaas// (page 9, 10)
cc/
[22] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL https: : htt//oppreeieeeeeeeee wwwpen/wwwwwwwweeeeeeeeeeeeeeeee formm//rmm http:/orruundddPoPPoo.tCt..
(page 10)
[23] Arash Nasr-Esfahany, mohammadIman Alizadeh, and Devavrat Shah. Counterfactual iden- tifiability of bijective causal models. ArXiv preprint, abs/2302.02228,, 2023. URL https:
/ //arxiv.org/abal ats//////////////////////a/e//// http:////aaaaaa././2222.. 230200222..222222 (page 1, 10)
[24] George Papamakarios, lain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2338-2347, 2017. URL https: http://///pooceeein.eeeeerr ips.cci..cc. cct..ccccppooe pappp//01//haaeeeeeeppppppaapaee (c11a886622c668222bbe www.2000000000000f00eea wtwp0o0099faaabstaatttteteeeeeee
(page 3, 8, 27, 28)
[25] George Papamakarios. Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22:57:1-57:64, 2021. URL http::/jmmlrr orgge/ggg/ee.oo http//ppprss///// http//////1//1/10 10m..html. (page
6. 16)
[26] Álvaro Parafita and Jordi Vitrià. Estimand -agnossic causal query estimation with deep causal graphs. IEEE Access, 10:71370 2022. doi: 10.1109/ACCESSS2222..3888
(page 1,
2)
[27] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: //tooced//p
neur neur pttppppppppppppppppppppppppppppp cc/paper/0020///////ecoo
(1777668fcc0bbcc8888888222222220 abb.:922211-btttttt.....t.a (page 2,
[28] J. Pearl. Causality. Cambridge University Press, 2009. ISBN 9781 13964399949 URL https:
/ //books //bboks.ooggle.ooo
(page 2,
[29] Judea Pearl. Causal inference statistics: An overview. 2009. (page 8)
[30] Judea Pearl. The do-calculus revisited. In Proceedings of the Twenty-Eighth Con- ference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, August 14-18, 2012, pages 3-11. AUAI Press, 2012. URL https://tslpii..org/nas 2:mm://mmmmmmmmmmmmmm2aaaa
28. (page 6, 21)
[31] Murray Rosenblatt. Remarks multivariate transformation. The Annals of Mathematical Statistics, 23(3):470-472, 1952. ISSN 00034851. URL httt::///www.sttorrrg////ee stabllllle/ stable/ 2236692. (page 4, 18)
[32] Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation.
In CLEaR, 2022. (page 2)
[33] Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series (Methodological), 13(2):238 -211, 1951. (page 8)
[34] Pablo Sánchez-Martin Miriam Rateike, and Isabel Valera. Vaca: Designing variational graph autoencoders for causal queries. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7):8159-81) -168, Jun. 2022. doi: 10. 10.1609/aaai, v36i7. 20789. URL https: haapps:////ssaaaaassssss...sssss woo.ssaaai.o.................... (page 2, 5, 6, 8, 9, 23, 27, 28, 29)
index.php/ Btteex.pp/AAAAT//A www.///atieeeeveeeeeeeeaeaeeeeee
[35] Robert Tarjan. Depth-first search and linear graph algorithms. SIAM Journal on Computing, 1(2):146-160, 1972. doi: 10. 10.1137/0200000. URL httpp::/ddo org http://oi...rrr1111///0000000000 http:/...13//0200011.
(page 4, 17)
[36] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervieed learning with data augmentations
12

---

provably isolates content from style. In Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December
6-14, 2021, virtual, pages 16451-16467, 2021. URL http:pp:s//proocccciig.eeeeeeeee paper ottp//////////oooooe/eeeeeeeeee0
(page 5)
[37] Julius Von Kügelgen, Abdirisak Mohamed, and Sander Beckers. Backtracking counterfactuals.
In Conference on Causal Learning and Reasoning, pages 177-196. PMLR, 2023. (page 7)
[38] Quanhan Xi and Benjamin Gloem-Reddyy Indeterminacy in generative models: Characterization and strong identifiability. In Proceedings of The 26th International Conference on Artificial Intel- ligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 12- 6939, PMLR, 25-27 Apr 2023. URL https: http://////oceenngs..illo.. wtw.://ss.....ess////eeeeeeeseee /v2sss///22////// 101006/xx2aaahheem html. (page 15, 16)
[39] Kevin Xia, Yushu Pan, and Elias Barcinboim. Neural causal models for counterfactual identific- ation and estimation. ArXiv preprint, abs/2210.00035,, 2022. URL https: /ttpa/////////////////////oo abs/ 2210.00035. (page 1)
[40] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning neural structural causal models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual, June 19-25, 2021, pages 9593-9602. Computer Vision Foundation IEEE, 2021. doi: 10. /0Wp/C/VRR644472222.00000 (etr2o22/atmmm/aag_Cauusss Seuraa__ttuctuura___as_________
hwt.:cote//tttto////o/eo/mm URL oanttaaaaaaeeeeeeeeeesaaaroaaooa weestaaggeeddeeeeeeeeeeeeeeeeeee SUTFR_2221_paper.htaa (page 1)
[41] Matej Zečević, Devendra Singh Dhami, Petar Veličković, and Kristian Kersting. Relating graph neural networks to structural causal models. ArXiv preprint, abs/2109.04173, 2021. URL
https:/ http://axvvvvor...e. www..agg//////.ooo///../oe..ee// http:/bs//10000 04173. (page 2)
13

---

Appendix
Table of Contents
Theory causal normalizing flows Identifiability results Extension real-world settings
A.1
15
A.2
15
B The multiple representations of SCMs
19
B.I Illustrative example
19
B.2 Non-linear SCM representations.
20
C Do-operator: inferventions and counterfactuals
21
C.I Definition and algorithms
21
C.2 Interventions in previous works
23
D Experimental details and extra results
24
Ablation: Time complexity
Coue............................
26
D.2 Ablation: Base distribution
Ateoa......l....................
27
D.3 Ablation: Flow architecture
27
D.4 Comparison: Extra non-linear SCMs
28
E Details on the fairness use-case
29
14

---

A Theory of causal normalizing flows
A.1 Identifiability results
First, we provide a more detailed explanation on the connection between the results from $3 and those from the work of Xi and Bloem-Reddy [38]. We consider it important to clarify that the definition of identifiability that we use is the same as [38, Def. 2]. Specifically, this definition one better suited for deep learning models, which is concerned with recovering the variables and one parametrization that perfectly matches the original generator. In other words, with this definition we aim to recover one parametrization of a neural network which provides the generator function, but not the exact parametrization of the generator that generated the data.
We also want to clarify that Thm. 1 from the main paper corresponds to [38, Prop. 5.2], which we rewrote (without changing its content) to plain English and match our particular setting. We now provide the proof for Cor. 2:
Corollary 2 (Causal consistency). If a causal NF To isolates the exogenous variables of an SCM M. then VxTo(x) = A and Vu (u) = + >nen )ddim(4) A", where A is the causal adjacency matrix of M. In other words, Te is causally consistent with the true data-generating process, M.
Proof. Assume that have flow Te that does indeed isolate the exogenous variables, meaning that the i-th output of the flow, ( is related with the true exogenous variable, uy by an invertible function that only depends on it.
As explained in $3, this means that for a variable u ~ PM. we have that To(((u)) ~ Pe and To (f((())) = huu)= = (hi h2 ... ha(ua)).
But we know the true generator, whose i-th exogenous variable is given by u; fr f x:) (the inverse of f. w.r.t u,) and, putting all together,
Tx(x); hu, = 14; (xpa, x:)):
which is function of only the parents of x; and x; itself.
If we call fx) = (Xpa, x1), f2 (xxpa X2):. ..... Sā (Xpa - xa)) the inverse of the SCM M that writes u as a function of x (see App. for an example). then clear that
To(xx (h f (x)) Vuh(u). vxf D Vxf =I A, (10)
where D is diagonal matrix and A the adjacency matrix of the causal graph induced by M. Similarly, To (h(()) = x= = (u) and TØ (h(u)); = x; = fi( u:), which again implies that:
diam( ())
V. (I -)- -= I+ Σ A"
(11)
where we call = h(u) the variable u recovered by Te, and where we have omitted as its Jacobian matrix is diagonal. Note that the infinite sum above vanishes at n diam A since A is triangular with diagonal zero.
E.E.D. D.
A.2
Extension to real-world settings
A.2.1
Discrete data
In this section, describe how extend the results presented in the main text for the case where one observed variable, x;, is discrete. To this end, we restate the more general data data-generative process assumed by Xi and (ldem-Reddyy [38], which we used for the theoretical part of the manuseript.
Following the notation of the manuseript, say that we have a data- eetta-genraatngg process without recursions, that is, have function f that maps x. Let us assume, without loss of generality, that only the i-th observed variable is discrete, and let us focus on the way this variable is generated, dropping the subindex along the way to avoid clutter. Now, Xi and Bloem- Bloem-Reddy Reddy [38] additionally
15

---

consider the existence of fixed noise distribution P. and mechanism 9, such that the observed variable X; is generated as,
(U1, U2................... ................................ UP)
Pu
E Pe.
g (un: u)),),
(12)
with E uul, and where they study the noiseless case under the following assumption: if for two generative processes with en Eb. then g(f((u)), = g(f (u,), if and only if fi(u)) = fi(w)).
where _ denotes equal in distribution.
Just as we do with the rest of variables, we also make the assumption that the observed variable x, is the transformation of a continuous exogenous variable, u, with a function ft that fulfils our assumptions (i.e., that ft is a diffcomorphhsmm) that has undergone a quantization process, i... x; = fu u,) = (u u. Therefore, is clear that fi is no longer bijective, as we are clamping real numbers into integers, and that the observational distribution of x; is discrete.
We take advantage of the noise assumption above, and dequantize the observed variable x, by assuming an additive noise mechanism such that = fi (uan. u,) + E, with distributed between the unit interval with any continuous distribution (we take in our experiments P. ((, 1)). With this process: i) we have made & again a continuous random variable, as the sum of independents discrete and continuous random variables a continuous random variable; and ii) the original distribution of the noiseless observed variables is always recoverable P(x; = c) = P(c S x; S c 1).
More importantly, all the theoretical insights from the work of Xi and Bloem- Reddy [38] can still be used, working with the noisy case rather than the noiseless one. Indeed, as for their analysis they assume single in the domain of the generator, we can merge the generator and noise mechanism go R-- + R (rather than go fa IR x [0, 1] + R), by mapping the non-injective part of u to & itself, i... by using the function (g o fi) Uan, u.) = (u u;)] + (f u,) - (un,. l) where F-1 is the quantile function of P.. This new function is diffeomorphism almost everywhere, as it is a composition of a.e. diffeomorphisms, and we have effectively replaced the noise variable by the floating part of f. (ua,, u,) before quantization. Moreover, note that if the noise is uniformly distributed, Pe- = (0, 1), we have that o fa = J. (if x; were discretized by taking its integer part).
In short, by adding noise to discrete variables while keeping them recoverable, we can learn a mapping between continuous variables that learns a version of the generator function before the observed values were somehow discretized. Importantly, the observed discrete distribution is always recover- able, independently of whether we learn the (unknown and unrecoverable) underlying continuous distribution before being discretized.
A.2.2 Partial knowledge
In this section, we explain how to expand our framework to settings in which we have partial information about the causal graph of M. That is, we know the causal ordering (so we know that half of the causal relationships, the upper diagonal), and we are certain about some other causal relationships (edges on A), but not all of them.
To this end, first let us first introduce the way we deal with partial knowledge, and then clarify the theoretical implications that it has with respect to the theory introduced in $5.
The method Similar to $4, let us motivate the method with an illustrative example. Suppose that we are given an SCM such as the one in Fig. 6a, where we know all relationships but the one between X2 and X3. Note that, in this case, we lack even information about the causal ordering. Indeed, there are three possible outcomes: i) the edge X2 + X3 could exist (Fig. 6b); ii) the edge X3 + X2 could exist (Fig. 6c); or iii) both could exist simultaneously (Fig. 6d), and hence there is a confounder between them. However, we do not know which of the three options is the correct one.
Let switch now Fig. 7. To solve the original problem (Fig. 7a, one natural approach is to group the nodes with unknown relationships assuming that all unknown edges may exist- and maximize the observed likelihood (Fig. 7b). This, effectively, is equivalent to applying an ANF to the known relationships. and using a However, if want to keep using exclusively ANFs (Fig. 7c), we can learn the joint distribution within the blocks with an ANF using a fixed ordering (which it can always do, as it is a universal density approximator [25]). The only subtle detail here is that, in that case, we need to increase the
Jacobian NF to learn the joint of the block variables.
16

---

0
0
0
0
0
0
0
0
?
0
1
1
1
0
0
1
0
0
0
0
0
0
Partial knowledge.
(b) Option 1.
(c) Option 2.
(d) Option
Figure 6: Example of an SCM with partial knowledge about the causal graph (a) and possible outcomes: (b) in the actual SCM only the edge X2 + X3 exists; (c) only the edge X3 + X2 exists; (d)
both edges exist (and therefore there exists a confounder between them).
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
1
1
0
0
0
0
0
0
1
0
X3
X2
(a) Partial kntileeee SCM.
(b) Solved with general NE.
(c) Solved with an ANE
Figure 7: Illustrative example (same as in Fig. 6) applying our method for partial information. First, we apply Tarjan's s algorithm [35] to find the SCCs of the graph (rectangles) and build a new DAG where each node subset of the original nodes. If, for the SCCs, use an NF with a general Jacobian matrix (b), we keep the individual edges and treat each SCC as a block. If we instead use an ANF (c), pick an arbitrary order within each SCC, and merge the individual edges into SCC-wide edges, Red represents intra-SCC edges, and green inter-SCC edges. See App. A.2.2 for more details.
granularity of all inter-block edges from node- to block-wise relationships, assuming that an edge exists if it exists for at least of the elements of the block. To why this is necessary, assume that the real graph of the example is Fig. 6c, yet we use an ANF with the ordering π = (1 2 4) and the graph A without inter-block modifications (i.e., the adjacency matrix of Fig. 6b). In that case, would have that X4 depends on X3 through x2. However, a causally consistent NF w.r.t. would not be able model that dependency, and thus x4 would depend on u1, U2 and uy but not Ug. With this approach, the ANF can model every case from Fig. 6, and it would need to remove the extra spurious relationships through optimization.
Therefore, to reuse our existing results from $5, the method that we have adopted the one from Fig. 7c, which can be described in the following steps:
Run Tarjan's algorithm [35] to group all nodes by their SCCs (note that, unlike the given example, there could be more than one cluster of unknown relationships).
2. Choose ordering that is consistent with the known inter-SCC edges, fixing the edges within the SCCs.
3. Move from node-edges to SCC-edges, In practice, this means introducing edges between every pair of edges of two SCCs, if there exists at least one edge between them. Solve the MLE problem with an ANF as in the main manuscript.
The theory Very conveniently, the method described above fits almost- pernestt-erfeetly into already- covered theory in $3. To see this, note that our identifiability theory and following results required a fixed ordering, but it does not need be the causal one: we can always find an equivalent SCM following the selected ordering by applying KR transports described in $3. Therefore, the technical
17

---

implications of both Thm. 1 and Cor. 2 still apply to this fixed ordering, we only need to re-state their implications with respect to the true causal genterannga process M.
To this end, we only need note that all possible graphs, once reduced into DAG using the partition of SCCs (as in Fig. 7c), are exactly equal. In other words, every possible graph shares the same causal dependencies befween SCCs with the other possible graphs. If we start treating them like block, calling {xi) C St C (1,2..... df the SCC of the i-th node, and påg, and ans; the parents and ancestors of every node in the SCC S, then it is clear that we can write for every graph A write its "exogenous" variables as a function of the true ones, i.c., üs = (R(us,) ). where KR is the Knöthe-Rosenblle transport [19, 31J. Note also that it does not depend on the data, i.e. üs _lx(us.
its observed variables as X; = (xps,, üs,) and, more importantly,
Theorem 3 (Identifiability Partial knowledge). If an element of F x Pu. and another from F x Pu. where F and F are TMI maps with different intra-SCC orders (see above), generate the same observational distribution, then the two processes differ by an invertible, SCC-wise transformation of the variables u.
Proof. Call M and M the elements from F x Pu and F x Pu, respectively.
W.L.o.g. pick M, and apply KR transport write it down another element of F Pu, call M, with identical observational distribution both M and M.
Using Thm. know that the elements of M and M differ by an invertible, component-wise transformation h. Moreover, we can write ús as a function of us,, as argued above, where {i} C S (1.2.....de d) are the indexes of the SCC that contains u.- Putting it all together:
u; h, (ū,) = h4 (KR,())))),
(13)
and, in vectorial form, for each SCC S.
üs hs, (KRs, (us.) (hs, (us.)
(14)
Q.E.D.
Corollary 4 (Causal consistency - Partial knowledge). If causal NF To, with partial knowledge of the causal graph, SCC-wise isolates the exogenous variables of an SCM M, then To is causally consistent with the true data-generating process, M, with respect each SCC.
Proof. The proof is identical to the one for Cor. but using arguments with respect the reduced graph after grouping all nodes in their respective SCCs.
Specifically, we can write using Thm. 3 the output of the flow as a function of the exogenous variables, (xxx); h(us,), and using the true causal generator, have
(x(x); h(us,) h(fs, (Xps, xs.)),
(15)
and hence the gradients agree with those from M, when looking at the reduced graph.
D.
Thm. 3 and Cor. 4 provide analogues to those results from the main manuscript. It is important to note, however, that causal consistency refers to the causal relationships between SCCs as a whole, i.e., not the causal relationships between individual nodes. The reason behind this is the same why we had to introduced spurious edges in Fig. 7c: as we fix an ordering within each SCC, we may not be able to model indirect dependencies of nodes of one SCC to another unless shortcuts. As such, every result from the main paper holds, if we treat each SCC (or block) as a whole. That is, when reason about SCCs instead of individual nodes (note that if the whole causal graph known every SCC contains a single node), we can safely talk about SCC-identifiability, causal SCC-consistency, and we can perform interventions and compute counterfactuals on SCCs.
artificially introduce
18

---

Gx Ju
G3(Ga(G,))
x= +G++++J
u= (I G)x
X
u
x
x
(a) Recursive.
(b) Unrolled.
(c) Compacted.
(d) Inverted.
Figure 8: Example of the linear SCM (xy==u x2== + +u) written (a) in its usual recursive formulation; (b) without recursions, with each step made explicit: (c) without recursions, as a single function; and (d) writing as function of x. The red dashed arrows show the influence of b1 on X3 for all equations from u to x, with the compacted version exhibiting shortcuts (see $4). Note that in the linear
have A G o 0, and that G1, G2, G3 G I any three matrices such that their product equals G + G + I.
B
The multiple representations of SCMs
B.1 Illustrative example
In this section, we delve bit deeper into the illustrative example from the main paper (64), and write down the maths understand how are these different models equivalent. For convenience the reader, we have replicated Fig. in the appendix as Fig. 8.
Recursive SCM
explained in the manuscript, the usual way of describing an SCM M is by providing its recursive equations. In our example (Fig, 8a), we have a linear SCM of the form
X1 i1
X2 2x + U2
(16)
X3 3x2
+ Uy
which we can compactly write as x = Gx + Ju. However, the recursive equations are not the most convenient ones, as they entail solving the system iteratively according to its causal dependencies.
Unrolled SCM Instead, we can write the equations function from u to x directly. To do this, we can proceed and unroll the equations:
X1 U1
X1 i
X1 = Ul
X2
2x + U2
X2
2u; + U2
X2 2u + U2
(17)
3x2
+
3(2x1 + U2) +
Us
X3 = 3(2u + U2) + U3
which we can write as multi-step function:
2 u1
2 = 2
X1 zi
2
2 U2
23 2z
+ 21
X2 22
(18)
7/3 Ug
7/3 7/2
X3 = 373 + 2
that we can once again compactly write as a series of linear operations x = G (G (GGzzGuul ())).. Note that the matrices G1, G2, G3 are not unique, and that they are valid long they most as sparse as G, and produce the same final output.
Compacted SCM Another natural step here is to compress this sequence of linear equations into single operation. That is, use directly the linear operation described at the end of Eq. 17:
X1 U1
X2 2u U2
(19)
X3
6u;
3u + U3
19

---

And we can derive the same result in vectorial form:
x Gx -u= x G(Gx + Iu) + Iu = x= -G Iu) + Tu) + -u +
x= 6 G
Gu Iu = (G² + G T)..
Unfortunately, as described in $4, in this form we cannot longer distinguish indirect paths: we have collapsed all paths into direct paths. Even worse, if there were more than one path between two given nodes, we have combined their contributions into single path, making quite difficult to disentangle. This effect can be seen as analogous to the common process in cryptography for sharing secrets: if you have two primes (paths), it is fairly easy to multiply them and obtain their product. but if you have their product (collapsed paths), performing prime factorization of the number is prohibitive.
Inverted SCM Finally, we can take any of these different representations and invert the equations go from x to u. In this case, easier to work with the original equations:
Xi u
U X1
X2 2x1 + U2
U2 X2 2x1
(20)
X3 3x2
+ Ug
03 X3 -332
and, in vectorial form:
x Gx + Iu= G)xx
(21)
As discussed in the main manuscript, this turns out to be a really convenient SCM representation to work with, can obtain the exogenous variables in one go while being causally consistent.
B.2 Non-linear SCM representations
In this section, we diseuss how the same representation and reasoning about the causal relationships of linear SCM from App. B.1 can be to the general case.
To this end, assume that we have a non-linear SCM M of the form f(x, u) where, to ease the reader, imagine that it has the same causal graph as the linear example, so that the reader can use Fig. 8 as a reference again, i.e., assume that
0 0 1
0
(22((x + 0) =
and (v) + +0) =
1)
(22)
0/
0
0
where 0 is the constant zero function with the same domain and codomain as the Jacobian matrices.
Recursive SCM Already extensively discussed. This is the representation an SCM is given as.
Unrolled SCM Just as before, we can unroll the equations by having multiple functions z' f((z'=1), and each one we unroll those equations for which we already know the non-recursive equation of its parents, leaving all the other fixed (identity functions). As before, we can write these multiple layers in different ways, long as they produce the same final function (after composing them, f, f2 o ....o o f,, = f), and that they respect the causal dependencies provided by + A. This similar the linear example, and what we used to design the generative model in Eq. 5 of $4.
Collapsed SCM Just before,
expand all the different layers and write (probably complex) formula that encompasses all changes in a single step. is easy to show that the composition of these
functions has, in general, Jacobian matrix structurally equivalent to I + > diam((A A Specifically, their composition will be of the form [I. + A).
Since we assume that each (xpa, u,) is bijective with respect to u, we can always compute its inverse to obtain u; as a function the observed values, u, = f xz). Clearly, its Jacobian matrix will be structurally equivalent to I + A = I A.
Reverse SCM
Therefore, always reason as we did with the linear case, but using Jacobian matrices talk about causal dependencies between variables, and possibly having a complex and/or non-closed formulation of the generatiie/aadducivv functions.
20

---

C Do-operator: interventions and counterfactuals
C.1 Definition and algorithms
In this section, we extend on the do-operator implementation described in $5, and provide the step-by-step algorithms to perform interventions (Alg. 1) and compute counterfactuals (Alg. 2).
Semantics Recalling $5, the do-operator [30], denoted do(x; = o.), is defined as a mathematical SCM M, inducing an alternative model M
operator that simulates physical intervention that fixes the observational value x, a, and thus removes any causal dependency on X5. However, the definition does not describe the specifics on how to implement such operation.
Usual implementation Traditionally, we are given the recursive representation of an SCM (Fig. 8a), as discussed in App. B. As such, the do-operator do(x; ax) is usually carried out by replacing the i-th equation, i.e., the i-th component of f, with constant function. That is, by doing f a. This yields an intervened SCM M = (F-, P.) reflecting the data-generating process after such an intervention. Unfortunately, this implementation of the do-operator is quite specific to the recursive representation of the SCM (Fig. 4a), and does not translate well to the other equivalent representations discussed in $4 and App. B. The reason for this is that these representations compute the observational values x as a vector function of u, without the iterative sampling process that goes through the intervened value that we replace. The only exception is the abductive model with knowledge on the causal graph and I. (see $4), as corresponds to the exact same case of Fig. 8d. If we have whether this implementation works or not comes down to the specific implementation to compute the inverse of the network.
however,
Proposed implementation As discussed in $4, we instead propose to manipulate the SCM by modifying the exogenous distribution P.. while keeping the causal generator f untouched. Specific- ally, an intervention do(x; = a) updates Pn. to have positive density mass on only those values that, when transformed to endogenous variables, the intervened variable yields the intervened value, x; = a, while keeping the rest of distributions unaltered.
In other words, we define the intervened SCM as M = (f, Pi),, where the density of the updated distribution PJ is of the form
p (u) x p(u) ((tu).((100 (
(23)
and where the distributions of the rest of variables remain the same. Using the acyclic assumption, we know that the only way of altering the value of x, without altering those of its parents is through u, and, using the causal sufficiency assumption, we can squeeze the Dirac delta directly in the distribution of the i-th exogenous variable, such that:
(u = po (u:l II(;) with pf (u:luzz4) x pr(us) . (1) jut
(24)
In the case consider, where all the generators are bijective given the parent nodes, the set ([(e. I (u) contains a single element, and therefore in the main paper we simply write the i-th density as p. (u, (y = ( (u). Note, as discussed in the main paper, that the density at this point should be positive, in other words, the element that yields a (and therefore a) should be plausible value.
Since this implementation does not make any assumption at all in the functional form of the generator, but directly works on the distribution of the exogenous variables, it can be implemented on any SCM representation (see Fig. App. B. and, hence, on any the architectures for the causal NF in $4, Notice, however, that in order to properly work, that the enta-generative process should be causally consistent (so that changes are properly propagated), and that it should properly isolate u (so that u; accounts only for the stochasticity of x; not explained by its parents).
The step-by-ster algorithms to perform interventions and compute counterfactuals, using the described algorithm, are presented in Alg. and Alg. 2, respectively. It follows the same description as the one given at the end of $5, and the only difference between both algorithms is the way that we obtain samples from the observed distribution (generated vs. given).
Algorithms
21

---

Algorithm 1 Algorithm to sample from the interventional distribution, P(x do(x; = ar)).
1: function SAMPLEINTERVEEEDDIISTI. a)
2: u Pu
3:
To (u)
Sample value from the observational distribution. & Set x; to the intervened value a. Change the i-th value of
4: X; f- a
5:
u, (x);
6:
x Tø: (u)
7:
return x
> Return the intervened sample.
8: end function
Algorithm 2 Algorithm to sample from the counterfactual distribution, P(x*f do(xx x))x)).
I: function GETCOUNTERFACTUALIC/ a)
2: u
Tx(x²)
Get u from the factual sample. b Set x; to the intervened value a. Change the i-th value of
3: xf ta
4: u, u Tx(x
5:
xf Tg (u)
6: return x
Return the counterfactual value.
7: end function
Theoretical results Here, we briefly discuss way this method works, i.e., why the proposed implementation removes every dependency from the descendants with respect to the ancestors that go through the intervened value, as it is not directly obvious.
To see why, take the usual recursive representation of an SCM in the illustrative example from App. B.1 (Fig. 8a), and assume that we do do(x, a),, where X2 2x + U2 in this example. By updating the density P2 (u2), we have basically fixed the value of U2 be the only one that keeps a given X1, i.e., U2 = dependency of X3 on X1, we get
2x, (this can be clearly seen in Fig. 8d). If we now compute the
d X3
0xg dx2
0x3
(xx2
Ox2
U2 Ou, xx
Oxa 0x2
(2 1- (-22) =0
(25)
d dx1
1x21x1
0xg
0x1
In layman's terms, the value of u2 is chosen such that it fixes the value of a, countering any influence that the parents could have on X2 (or any of its intermediate values), and consequently in any of its descendants.
The general case can be similarly proven. Suppose that we do do(x2 = a). and that we want to compute the indirect influence of an ancestor, X1, descendant, X3, passing through x2. Since are fixing the value of u2 (the input of the network) to produce an observed value X2 (the output of the network) of x1) on X2 via U2:
use implicit diffcrentiation compute the influence of (and therefore
X2 (U1 U2)
dus,
Ox2
Ox₂ d du
Ox2 duz
8xg
(26)
Ou
Oug duj
Ou duj
Ou
and, similar to Eq. 25, any indirect influence of the ancestor, ui on the descendant, x3, through this intermediate variable, X2, cancels out:
dx3 dx2
0x3
Oxx2
0x2 du2
Ox3 Ox2
+
0 = 0.
(27)
Ox2 du
Ox2
ou
Ouz duj
We have proven this result not only theoretically, but also empirically through the accurate inter- ventions computed for all the experiments from $6 and App. D. Moreover, this lack of correlation between ancestors and descendants through the intervened variables is clearly shown in pair plots such the ones in Figs. 12 and 13.
22

---

C.2 Interventions in previous works
We now put our implementation of the do-operator (see $5 and App. C) into context, by describing how the methods compared in $6, namely CAREFL [16] and VACA [34], proposed to perform interventions with their models.
CAREFL [16] Two different algorithms were proposed sample from an interventional distribu- tion in CAREFL: i) sequential algorithm which mimics the usual inplementation of the do-operator with the recursive representation of the SCM: and ii) parallel algorithm that samples the counter- factual in a single call. While the first algorithm works, the parallel one one-which is the one actually implemented-onl, works when intervening on root nodes.
This second algorithm for do(x2 = a)) is described as follows (see Alg. in [16]): i) sample from Pu: ii) set to the i-th value obtained by applying the flow, To, to an observation with X; a and Xy 0 for j A i; and iii) return the value obtained by Tg with u.
While this algorithm resembles the one we proposed, the proposed method does not have into account that the value of u, fix does depend on the observed values its parents, which clear by looking at the linear illustrative example from Fig. 8d. As a consequence, the algorithm only works when the node has no parents, which is why we replaced by the one we proposed for the comparisons in $6 and App. D.
VACA [34] Based on GNNs, the approach for intervening on VACA is completely reminiscent to the traditional implementation. Specifically, the authors propose to sever those edges in every layer of the GNN whose endpoints fall in the path generating the intervened variable, so that the ancestors have no way to influence it by design.
While the previous statement is true: ancestors cannot influence the intervened variable nor its descendants, here we argue that this process would require "recalibrate" the model, as the middle computations after an intervention change in more complex ways than removing the ancestors from the equation, while keeping the rest unchanged.
To this, consider the following non-linear triangle SCM:
X1 = U1
X2 U2
(28)
X3 2x1
Ug
which VACA could learn with two layers through the following operations:
2i
Xi Z1
7g U1 U2
X2 7,
(29)
Z3 U1 + U2 + Ug
X3 = Z1 + Z2 + Z3
Now, if we were to intervene with do(x; a), the real SCM would yield:
X1 = U1
X2 a
(30)
X3 2x1
+ Us
x
while VACA would yield:
71 =
X1 = 71
X1
Zg
X
X2
(31)
Z3 U1 +a+ux
X3 Z1 + a Z3
X3 2x1 + 22 Ug
where we can clearly see that the expression for X3 is different. In contrast, our causal NF would keep the generator as it is, and set uy to a/x}, yielding the correct value.
23

---

D Experimental details and extra results
In this section, we complement the description of the experimental section from $6, and provide the reader with additional results in the following subsections. First, we describe the details common to every experiment, and delve into the specifics of cach experiment in their respective subsections.
Hardware Every individual experiment shown in this paper ran on a single CPU with 8 GB of RAM. To run all experiments, we used a local computing cluster with an automatic job assignment system, so we cannot ensure the specific CPU used for each particular experiment. However, we know that every experiment used one of the following CPUs picked randomly given the demand when scheduled: AMD EPYC 7702 64-Core Processor, AMD EPYC 7662 64-Core Processor, Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz, or Intel(R) Xe)n(R) CPU B5-2690 v4 @ 2.60GHz.
Training and evaluation methodology For every experiment, generated with using synthetic SCM a dataset with 20 200 training samples, 2500 validation samples, and 2500 test samples. We every model for 1000 epochs, and the results shown in the manuscript correspond to the test set evaluation at the last epoch. For the optimization, we used Adam [17] with an initial learning rate of 0.001, and reduce the learning rate with decay factor of 0.95 when it reaches plateau longer than 60 epochs. For hyperparameter tuning. we always perform grid search with similar budget, and select the best hyperparameter combination according to validation loss, reporting always results from the test dataset in the manuscript. Every experiment repeated 5 times, and we show averages and standard deviations.
Datasets This section provides all the information of the SCMs employed in the empirical evaluation of $6 of the main paper, and the following subsections. The exogenous variables always follow a standard normal distribution (0, 1). except for LARGEBD, where uniform distribution (((0, 1) is used instead. Subsequently, we define the 12 SCMs non-linear equations and additionally provide their causal graph in Fig. 9.
cothoodddddnoompssinnn... both linear and
Let us first define the softplus operation as s(x) = log (1.0 + e*).
3- 3-CHAINLaX:
(ui) =
(32)
fx(x1, ug) 10 X1 U2
(33)
fa Un) = 0.25 X2 + +--00 U3
(34)
3-CHAINNLIN
[u;) =
(35)
f2(x1, U2) 2/2 U2/4
(36)
Fa(X2, Uz) (x2 5) 3
U3
(37)
15
4-CHAIN
fu())) = U1
(38)
f2(x1, (2) U2) 51 U2
(39)
fx Ug) -0.5 X2 1.5 Ug
(40)
[x u4) = -33+
(41)
5-CHAIN
fu())) =
(42)
fe(x1, Un) = 10 X1 U2
(43)
[x(x,, Un) = 0.25 X2 + +2-05 Ug
(44)
fi(xs, un) = +
(45)
24

---

fs( = Xx1 + U5
(46)
COLLIDERIIN
(ui)
(47)
fu(u2) U2
(48)
fa X2, u3) = =0.25 X2 0.5 X1 + 0.5 U3
(49)
F
(u)) = U1
(50)
f2(u2) = 2-12
(51)
fa(x1, X2, Us) -0.25 X2 1.5 X1 + 0.5 U3
(52)
Fa(xs, un) = X3 + +.225 41
(53)
FORKNLIN
(u() Uj
(54)
f2(22) Ug
(55)
4
fx1,1, X2, Us) = + 1e- 2
X + 0.5 U3
(56)
20
fa(xs.us) =
+ U4
(57)
1 0.5.3 x3
LARGEBD NLIN: Let us define
L(x, v) s(x 1) s(0.5 + +) 3.0,
(58)
and us call (p, .>) the quantile function of Laplace distribution with location μ4, scale evaluated at x. Then the structural equations are
(u)) s(1.8 un) -
(59)
f2(x1, U2) 0.25 U2 1x(x1, 1.5
(60)
fx(x, Ua) = L(X1, U3)
(61)
fx(x2, ua) L(x2, Un)
(62)
J5(xa, us) I(X3, us)
(63)
fo(x1, UG) = L(x1, U6)
(64)
fr(8s, U7) L(Xs, U7)
(65)
fx ug) 0.3 us + (s(X6 +1)
(66)
fo Xg. ug) CDF X7 1.3 + X8
+1 2.0.6 ug
(67)
SIMPSONSUNN
fi(u)) = U1
(68)
fe(x1, Ug) s(1 x1) + 33/20.. U2
(69)
Fs X2, Uz) tanh (2 X2) + 1.5 x1 + tanh(u3)
(70)
fx un) X3 4 +3
1
U4
(71)
V10
25

---

SIMPSONSYMPRES
fu())) in
(72)
1
f2(x1, u2) = tanh (2-x)) +
U2
(73)
v10
1
fx(x1, ua) = 0.5 112
(74)
V22
fx( = tan (1. x1) +
V
u4
(75)
10
TRIANGLE
fi(u;) = U1 +1
(76)
82( 10
(77)
fa(, U3) 0.5 X2 XI 43
(78)
TRIANGLENLIS'
fi(u1) = U1 +1
(79)
f2(x1, U2) = 2- x + U2
(80)
20
fa(x1, X2, Us)
+ U3
(81)
+2
T3
31
(a) 3-CHAIN
(b) 4-CHAIN
(c) 5-CHAIN
ST
J2
J1
122
(d) TRIANGLE
(c) COLLIDER
(f) FORK
x1
T2
J'
JEA
Z1
3.4
22
TA
(g) SIMPSON [NLIN]
(h) SIMPSON [SYMPROD]
(i) LARGEBD
Figure 9: Causal graph of the different SCMs considered in $6 and App. D.
D.1 Ablation: Time complexity
As the first additional ablation study, we evaluate the time complexity of the design choices introduced in $4. Fig. 10 summarizes the results, where the x-axis represents the number of nodes in the dataset,
26

---

Direction: X
Model:
Ordering
Ordering
Graph
-x-
Graph"
Training time (ps)
15
10-
5
Size of graph (d)
Figure 10: Time complexity comparison between the different design choices discussed in $4.
d, and the y-axis indicates the time (in microseconds) required for single forward pass of the normalizing flow during training. We can find several interesting insights:
Results First, abductive models (x + u, solid lines) train significantly faster compared generative models (u + x, dotted lines). This emphasizes the computational cost associated with inverting autoregressive flows during training. Furthermore, is worth noting that the time complexity of abductive models remains constant regardless of the number of nodes d, whereas the complexity of generative models increases linearly with as indicated the right-most most column of Tab Note that we should see the exact opposite behaviour when sampling.
Second, the inclusion Jacobian regularization, represented by star marker, introduces a significant time overhead. This observation is clearly depicted in Fig. 10, where plot lines with the star marker consistently above their counterparts.
Finally, leveraging causal graph information or relying solely on ordering (represented in orange and green, respectively) has minimal impact computational time. This is once again clear in Fig. 10, as green and orange pairs largely coincide.
D.2 Ablation: Base distribution
We now assess to which extent mismatch of the distribution Pu between the SCM and the causal NF negatively affects performance. To this end, we consider two more complex CCMSs-IMMPON [11] and TRIANGLE [34]-and Iitiibttins-Noomee and Laplace-for which we either fix or learn their parameters during training. Both SCMs use standard Normal distribution for Pa.
Hyperparameter tuning While we fixed the flow have a single MAF [24] layer with ELU [4] activation functions, we determined through cross-validation the optimal num- ber of layers and hidden units of the MLP network within MAF, Specifically, we con- sidered the following combinations (Ja, b] represents two layers with and b hidden units): [16, 16, 16, 16], [32, 32, 32), [16, 16, 16). [32, 32). [32], [64]. As discussed the start the section, we report test results for the configuration with the best validation performance at the last epoch.
The results, shown in Fig. 11a, reveal a notable distinction between Normal and Laplace distributions in terms of density estimation. However, this discrepancy appears to have minimal implications for Average Treatment Effect (ATE) and counterfactual estimation. We hypothesize that this disparity originates from dissimilarities in their tails, as can be inferred by the slight edge of Normal over Laplace on the last column column-which measures per sample differences where bigger errors happen at the outliers elements which are, by definition, scarce. Interestingly, with this particular architecture of causal NF, every model struggles to model the denser TRIANGLE SCM.
Results
D.3 Ablation: Flow architecture
Considering the observed challenges faced by the Masked Autoregressive Flow (MAF) [24] layer in accurately modelling the TRIANGLE SCM in the previous experiment, we further investigate the potential impact of flow architecture performance.
27

---

Daineet
SIMPSON [NLIN]
TRIANGLE [LIN]
Laplaces
0.5
MAF
0.3
Laphax
0.4
Nonns,
0.2
Normal
0.0
NSF
0.0
Bane Datr
KL forward
RMSE ATE
RMSECF
Model
KL forward
RMSE ATE
RMSE.CE
(a) Ablation on the base distribution
(b) Ablation the flow architecture
Figure 11: Performance on the SIMPSONNLIN and TRIANGLELIN datasets of causal NFs with different base distributions (Normal and Laplace), where 0 indicates that we learn the parameters of the base distribution; and b) flow architectures. Differences in base distribution affect KI. divergence, while the choice of flow architecture influences the overall performance.
Hyperparameter Tuning We cross-validate again the optimal number of layers and hidden units of the MLP internally used by the unique layer of the Causal NF, We consider the following values (la. b) represents two layers with and b hidden units): [32, 32, 32], [16, 16, 16), [32, 32), [32], and [64]. As before, test results are reported for the configuration that achieved the best performance on the validation set at the final epoch.
Results Fig. 11b summarizes our results, where we consider a Causal NF with one MAF [24] layer (depicted in orange), and a Causal NF with Neural Spline Flow (NSF) [9] as layer (depicted in blue). Note that, NSFs are built on top of MAFs and abandoned the realm of affine ANFs, and are thus expected to outperform MAFs in general. We employed the same set of SCMs as in App. D.2.
Our empirical analysis reveals that the NSF consistently outperforms the MAF across the three metries: observational distribution (measured by the KL divergence), ATE estimation, and counterfactual estimation. Whilst expected, these findings highlight the practical implications of selecting an appropriate flow architecture, which should be taken into consideration by practitioners.
D.4 Comparison: Extra non-linear SCMs
In this section, we complement the results from $6.2 and provide a more extensive comparison the proposed Causal NF, along with CAREFL [16] and VACA [34]. additional datasets.
Hyperparameter Tuning For VACA, cross-validated the dropout rate with values {0.0, 0.1}, the GNN layer architecture with {GIN, PNA, PNADisjoint). (see [34] for details), and the number of layers in the MLP prior to the GNN with choices {1,2). For CAREFL, the number of layers in the flow, {1, diam A}. and the number of layers and hidden units in the MLP composing the flow layers (same format as before), {[16, 16, 16), [32, 32). [32], [64]]. For Causal NF, we used the abductive model with a single layer, and cross-validated the number of layers and hidden units in the MLP composing the layer of the flow with values {16, 16, 16, 16), [32, 32, 32], [16, 16, 16), [32, 32), [32), [64]. We report test results for the configur- ation with the best validation performance at the final epoch.
cross-validated
Results Tab 4 shows the performance of each model for all the considered datasets, further validat- ing the conclusions drawn in the main manuscript: the proposed Causal NF consistently outperforms both CAREFL and VACA in terms of performance and computational efficiency. The performance of VACA is notably inferior, and its computation time is significantly longer, primarily due to the complexity of graph neural networks (GNNs). Our Causal NF achieves similar performance to CAREFL in terms of observational fitting, while surpassing it on interventional and counterfactual estimation tasks. Additionally, Causal NF outperforms CAREFL in computational speed. This is to be expected since the optimal CAREFL architectures often have multiple layers, resulting in increased computation time. In contrast, Causal NF has single layer, reducing computational complexity.
Fig. 12 qualitative proofs the effectiveness of the proposed Causal NF in accurately modelling both observational and interventional distributions for the SIMPSON dataset. In this plot, blue represent the real distribution/samples. while orange represents the ones generated by Causal NF. Fig. 12a clearly shows that the model successfully captured the correlations among all variables in the observational distribution. Furthermore, Fig. 12b displays the interventional distribution
28

---

Table 4: Comparison, on different SCMs, of the proposed Causal NE, VACA [34], and CAREFL [16]
with the do-operater proposed in $5. Results averaged over five runs.
Performance
Time Evaluation (us)
Dataset
Model
KL
ATERMSE
CFRMSE
Training
Evaluation
Sampling
3- 3-CHAIN
Causal NF
0.00p.
0.05g 01
0.04g
0.41o.
0.48p.10
0.76p.06
CAREFL
0.00g.00
0.200.13
0.200.00
0.68p.24
0.97o.33
1.940.77
[34]
VACA
4.44
5.760.07
4.98p.10
36. 19.14
28.33p.72
75.34
3-CHAIN
Causal NF
0.00a.00 00
0.03a.01
0.02a.m
0.52. oe
0.56g.05
1.02p.0
NLIN
CAREFL
0.00g.oo 00
0.05p.02
0.04g.02
6.600 22
0.84p
1.060.41
[34]
VACA
12.821.00
1.540.03
1.32o.u2
39 19.455.11
30.932.30
84.369.c
Causal NF
0.00p.00 00
0.07p.02
0.04g.m1
0.56g.os os
0.622.15
1.54p.40
4-CHAIN LIN
CAREFL
0.00
0.16p.07
0.14p.04
0.70
0.99p.20
2.850.54
VACA
13. 13.14
3.82o.01
3. 3.720.u5
61.85p.u6
49.31
92.067.9z
Causal NF
0.01p.00
0.12p.02
0. 0.08g.o
0.62a.19
0.69a.15
1.910.44
5-CHAIN LIN
CAREFI.
0.00g.oo
0.47o.23
0.466.22
0.79g.41
1.19p.25
4.2l
VACA
1..31 0.84
5.95o.05
6.06o.08
103.7510.04
80.8111.00
124.5220.s6
COLLIDER [34]
Causal NF
0.00g.on
0.02o.o1
0.01g.00
0.46g.12
0.56u.11
0.95u.19
CAREFL
0.00g.00
0.02p.oz o1
0.01
0.39
0.45g.es
0.74p.o7
VACA
13.45p.43
0.220.01
0.860.02
37 37.22g.55
28.774.22
71.21
FORK
Causal NF
0.00g.o0 00
0.03g.oz
0.01g.00
0.52a.o0
0.59p.c os
1.570.s7
LIN
CAREFL
0.00
0.04g.oz
0.02g.00
0.60
0.78p.16
2.391.00
[2]
VACA
8.75g.73
0.87p.02
1.430.02
45.84, 64
34.662.50 30
73.294.70
FORK NLIN
Causal NF
0.00
0.07
0.07
0.63g.
0.74p.s1
1.84p.84
CAREFL
0.01p.01
0.11o.04
0.18p.07
0.57
0.77
1.960.17
[2]
VACA
5.09p.60
2.01o.03
3.190.06
19.225.c
42.132. as
101.02 8.94
LARGEBD
Causal NF
1.51..04
0.02g.om
0.01
0.52a
0.60a.17
3.05p.ee
NLIN
CAREFL.
1.51
0.05o.01
0.08o.01
0.84p.c
1.18
8.251.29
[11]
VACA
53.662.0r
0.39p.00
0.82p.02
164.92 10
137.8815.12
167. 167.9423.75
SIMPSON [1]]
Causal NF
0.29g.01
0.04g.02
0.02g.o0
0.58g.1s
0.63g.26
1.570.04
CAREFL
0.29o
0.04p.01
0.03o.00
0.69o
1.02a.26
2.950.70
VACA
18.97..66
0.60p.00
1.19p.02
54.767.e 46.
43.69,
87.01
SIMPSON SYMPROD
Causul NF
0.00g
0.070.01
0.12g o2
0.59
0.60
1.510.30
CAREFL
0.00p.0 oo
0.100.02
0.170.04
0.49o. 15
0.81p.
1.91o.53
[11]
VACA
13.85p.64
0.89p.00
1.50p.04
49.20 49.26,
37.783.41
79.20.4.60
TRIANGLE
Causal NF
0.00g.00
0.240.05
0.21g.05
0.540.05
0.560.04
1.050.07
LIN
CAREFL
0.00n.00 00
0.15p.06
0.14g.03
0.60n 20
0.75p.05
1.50o.10
[34]
VACA
3.82p.60
7.490.07
7.220.17
27.46 .s.
21.611.00
67.00
TRIANGLE NLIN
Causal NF
0.00p
0.12
0.13g
0.52g.o0 07
0,58p o7
1.07o.12
CAREFL
0.00a.00 oo
0.12a.08
0.17o.ns
0.57 18
1.883 26
1.68p.e2
[34]
VACA
p.70
4.780.01
4.190.04
28.82,
23.00p.sr
70.65
obtained when we do do(x3 = 1109), , i.e., when we intervene on the 25-th empirical percentile of X3. Remarkably, Causal NF accurately learns the distribution of descendant variables, i.e., X4. and effectively breaks any dependency between the ancestors of the intervened variable and X1-
Additionally, Fig. 13 shows similar analysis for 5-CHAIN when we perform do(x3 2.18) which corresponds to intervening on the 75-th percentile of X3- xy-clearly clearly showing that the correlations not involving the intervened path (x1 X2 and X4 X5) are preserved.
E Details on the fairness use-case
In this section, we provide additional details on the use-case of fairness auditing and classification using the German dataset [8], whose causal graph is shown in Fig. 14.
Training For this section, we performed minimal hyperparameter tuning, and only tested a few combinations by hand. We decided use Neural Spline Flow (NSF) [9] for the single layer of the Causal NF, which internally uses an MLP with 3 layers, and 32 hidden units each. We use Adam [17] as the optimizer, with a learning rate of 0.01, along with a platcau scheduler with decay factor of
29

---

(a) Observational distribution.
(b) Interventional distribution do(xs = -1.09).
Pair plot of real (in blue) and generated (in orange) data SIMPSON On the left are samples from the true and learnt observational distribution. On the right are samples from the true and learnt interventional distribution when do(x3 1.09). The plot illustrates that the dependency of X4 the ancestors of X3, namely Xı and X2, is effectively broken.
Figure 12:
(a) Observational distribution.
(b) Interventional distribution dolx; 2.18).
Figure 13: Pair plot of real (in blue) and generated (in orange) data of 5-CHAINUx- On the left are samples from the true and learnt observational distribution. On the right are samples from the true and learnt interventional distribution when do(x3 2.18). The plot illustrates that the dependency of X4 and X5 on the ancestors of X3, namely Xı and X2, effectively broken.
0.9 and patience parameter of 60 epochs. The training is performed for 1000 epochs, and the results are reported using 5-fold coss-validation with 10 10 split for train, validation, and test data.
Results On addition to the results from $7, Fig. 15 shows two pair plots from one of the 5 runs, chosen random. The empirical distribution is shown in blue, and the learnt distribution by Causal NF is depicted in orange. Specifically, Fig. 15a illustrates the observational distribution, and Fig. 15b the interventional distribution, obtained when we intervene on the sex variable and to 1, i.e., do(xi = 1). We can observe that Causal NF achieves remarkable fit in both cases, demonstrating its capability handle discrete data, and partial knowledge of the causal graph.
30

---

checking
credit
savingy
repayment
history
age
housing
Figure 14: Partial causal graph used for the German Credit dataset [8]. Rectangles show the strongly connected components (SCCs) grouping different variables. Solid arrows represent causal relationships between SCCs, and dashed arrows represent arbitrary order picked to learn the joint distribution of each SCC with an ANF, See App. A.2.2 for an in-depth explanation on the proposed method to deal with partial causal graphs using causal NFs.
31

---

Observational distribution.
Interventional distribution do(xi
Figure 15: Pair plot of real (in blue) and generated (in orange) data of German dataset. Above, the samples from the true and learnt observational distributions. Below, the samples from the true and learnt interventional distributions when do(x) = 1). The plot illustrates that Causal NF is able to handle discrete data and correctly intervene.
32


Reviews:
[{"review_id": "fBiRWs7Tne", "reviewer": "Reviewer_1x9i", "summary": "This work explores the use of normalizing flows (NFs) in causal inference. The authors demonstrate that causal models can be identified from observational data using autoregressive NFs. They investigate design choices and implement the do-operator in causal NFs to answer interventional and counterfactual questions. The experiments validate their approach and show that causal NFs can effectively address real-world problems with mixed data types and partial knowledge of the causal graph.", "questions": "The authors should give the intuition of presenting Fig. 1 in a more clearly way.\n\nIt is well-known that computational complexity is a challenge when using normalizing flows, especially for high-dimensional data. In this matter, I believe that would be difficult to use the proposed method with such data (e.g. images). Is it limited to tabular data? Is it possible to use it with high-dimensional data? Any pre-processing steps are necessary?\n\nIt would be viable to learn the causal graph (e.g. PC) and use it as input the proposed method? My concern is whether it would be wast of time, given that the information is available in the data. In other words, is there a necessity of domain knowledge to use the proposed method?", "limitations": "The datasets used are not strictly representative of practical applications. I believe that there is an avenue of possible developments here turn the proposed method largely applicable.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The authors clearly stated the problem of interest as well as compare with relevant previously showed approaches. They also conducts extensive experiments, including ablation analysis. Finally, they discussed practical limitations and present possible lines of investigation as future work.", "weaknesses": "The number and characteristics of the given dataset are limited when compared to practical applications. The ablation analysis miss hyperparameter tuning.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for their valuable feedback, which points out details that we will clarify in the updated manuscript. Please see our responses addressing the specific concerns below:\n\n> The number and characteristics of the given dataset are limited when compared to practical applications. \n\nWe would like to point out that we test our framework in 12 synthetic SCMs (Table 4, App. D.4), including diverse causal graphs with both linear and non-linear structural equations (see Figure 9 in App. D). In addition,  we also use a real-world dataset, the German Credit,  which contains mixed-type data and partial knowledge on the causal graph. Our empirical evaluation is on par, in terms of number and complexity of the considered SCMs, with the one in VACA [[1](https://aaai.org/papers/08159-vaca-designing-variational-graph-autoencoders-for-causal-queries/)], and we believe it is sufficient to show the validity of our theory motivating design choices for causal normalizing flows. We agree with the reviewer that there is an avenue of future works  applying the proposed  causal normalizing flows to many application domains, also involving large dimensional data. \n\n> The ablation analysis misses hyperparameter tuning.\n\nWe are unsure if we fully understand the reviewer comment, but we would like to clarify that the main goal of our ablation study is to validate the claims with respect to the network design in Section 4. Regarding hyperparameter tuning, we test the choice of base distribution for the flow in Appendix D.2, as well as the impact of choosing a different flow architecture in Appendix D.3. Additionally, we provide details on hyperparameter tuning for all experiments in App. D.\n\n > The authors should give the intuition of presenting Fig. 1 in a more clearly way.\n\nThanks for the feedback. We indeed were over-concise in our explanation, and will update it with the following: \"This is exemplified in Fig. 1, where our proposed framework is able to estimate the (unobserved) causal effect of externally intervening on the sensitive attribute $s$, using solely observed data (blue distribution) and partial information about the causal relationship between features.\"\n\n> Is it limited to tabular data? Is it possible to use it with high-dimensional data? Any pre-processing steps are necessary?\n\nIn principle there is no limitation on the proposed causal normalizing flow that prevents us from applying it to high-dimensional data. That said, we would like to clarify that our proposed method is tied to the current limitations of normalizing flows and to the extent to which we can define a (potentially partial) causal graph. For example, NFs have been applied in image processing, but to the best of our knowledge are not the state of the art, and at the same time it does not seem sensible to build a causal graph at the level of pixels. However, one could consider using an image as a node in our causal NF, for example in medical applications where the image (e.g., X-ray) is one more dimension in the patient (potentially causal) data.\n\n> It would be viable to learn the causal graph? Is there a necessity of domain knowledge to use the proposed method?\n\nWhile the focus of our paper is not causal discovery, we believe that our work could also be extended for causal discovery. For example, one idea would be to integrate the proposed causal NF into the DECI  framework [[2]](http://arxiv.org/abs/2202.02195), i.e., perform causal discovery by modelling the joint distribution of both the observed data and the causal graph as $P(G, X) = P(G) P(X|G)$, where the causal NF is used to fit $P(X|G)$. Note, however, that DECI only applies to structurally identifiable SCMs, specifically to continuous non-linear additive noise models (ANMs). In general, it is not possible to estimate a causal graph solely from observational data, and either domain knowledge or additional assumptions are needed.\n\n---\n\n[1] [VACA: Design of Variational Graph Autoencoders for Interventional and Counterfactual Queries](https://aaai.org/papers/08159-vaca-designing-variational-graph-autoencoders-for-causal-queries/)\n\n[2] [Deep End-to-end Causal Inference](http://arxiv.org/abs/2202.02195)\n"}, {"review_id": "P2rkT9oMyj", "reviewer": "Reviewer_515Z", "summary": "This work explores the use of normalizing flows for causal reasoning. The authors demonstrate that causal models can be identified from observational data using autoregressive normalizing flows. They discuss design choices, learning strategies, and the implementation of the do-operator to handle interventional and counterfactual questions. Through experiments, they validate their approach, compare it to alternative methods, and show its effectiveness in addressing real-world problems with mixed discrete-continuous data and partial causal graph knowledge.", "questions": "I don't have any specific questions about the main paper, but I recommend that the authors consider transferring some crucial conclusions from the supplementary material to the main paper.", "limitations": ".", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: This work introduces a novel approach by employing causal normalizing flows to identify the underlying causal ordering and effectively address interventional and counterfactual queries using the do-operator. To the best of my knowledge, this methodology is both novel and reasonable.\n\nSignificance: The ability to predict the interventional effect of the causal data-generating process is a highly important problem with practical implications.\n\nContribution: The paper presents a clear and comprehensive method, accompanied by necessary conditions that support the theoretical aspects. The overall technical contribution is commendable.\n\nThe writing quality of this paper is good, and it provides ample experimental results to validate its effectiveness. I thoroughly enjoyed reading this paper.", "weaknesses": "Challenge: From my understanding, the previous work has already achieved satisfactory results regarding the identifiability of causal normalizing flows. This paper extends these findings to include counterfactual and interventional reasoning. While the theoretical contribution and methodology are commendable, I think this extension is relatively straightforward, and the challenge presented may not be substantial. Consequently, this could be considered the primary weakness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments, and we are flattered to know that the reviewer had a good time reading our work. While there are no specific questions, we would like to add a few clarifications below that we hope helps the reviewer better contextualize our work:\n\n> From my understanding, the previous work has already achieved satisfactory results regarding the identifability of causal normalizing flows.\n\nWe are unsure to which previous work the reviewer refers exactly:\n\n1. If the reviewer refers to the paper by Xi and Bloem-Reddy [[1]](https://proceedings.mlr.press/v206/xi23a.html), from which we adapt Theorem 1, note that their work involves identifiability of latent variable models in an ICA framework, where there is **no causal consideration.** Thus, our main contribution lies in the fact that by casting SCMs and ANFs as TMI maps, we can re-formulate causal inference (i.e., recovering the true exogenous variables)  as a blind-source separation problem, and thus leverage their identifiability results. To the best of our knowledge, our work is the first to provide identifiability guarantees for such a broad family of SCMs (lines 110-113). \n2. If the reviewer refers instead to the work that introduced CAREFL [[2]](http://proceedings.mlr.press/v130/khemakhem21a.html), there are a number of important differences worth pointing out:\n    1. The main focus of CAREFL is causal discovery (i.e., estimating the causal graph), while ours is performing causal inference (given a causal ordering/graph).  These are two related but different problems, each with their own challenges and applications.  Importantly, the main theoretical result of CAREFL regards only the bivariate case (each one being uni-dimensional) with Gaussian exogenous variables. \n    2. Their work assumes affine ANFs, while we allow for any type of ANF. For example, NSFs  [[3]](https://proceedings.neurips.cc/paper/2019/hash/7ac71d433f282034e088473244df8c02-Abstract.html) (which we use in the fairness use-case, Section 7) are non-affine ANFs.\n    3. In a similar note, they assume additive-noise SCMs, which is a subset of the bijective SCMs assumed in our work.\nThat said, we acknowledge that CAREFL indeed hints a connection between ANFs and SCMs, which in this work we: i) properly formalize, generalize, and make this connection precise; ii) provide general identifiability results; and iii) propose a network design to efficiently learn the underlying SCM.\n\n> I think this extension is relatively straightforward, and the challenge presented may not be substantial.\n\nWe respectfully disagree with the reviewer here. While we agree that the final design is straightforward (this is in our opinion a positive outcome of our work), the contributions and challenges should not be overlooked. Specifically, in addition to the novelty of our identifiability results (see our answer on relation to Xi and Bloem-Reddy [[1]](https://proceedings.mlr.press/v206/xi23a.html) above), we first show in Corollary 2  that causal consistency is a necessary condition at the global optima; and then, in Section 4, introduce the necessary conditions to design a  causally consistent ANF, which needs to: i) use the graph information, ii) have a single layer, and iii) be defined from $x$ to $u$. Any other of the possible considered networks will not be causally consistent by design, and thus may lead to poor estimates of interventional and counterfactual queries. Moreover, as acknowledged by reviewer azJz,  the proposed implementation of the do-operator is also non-trivial.\n\n\n> I recommend that the authors consider transferring some crucial conclusions from the supplementary material to the main paper.\n\nWe thank the reviewer for the suggestion, and we will make space to move important remarks back to the main manuscript. Reviewer azJz made a similar comment with specific content to add, and there may be some overlap. We have summarized the changes for the next revision in the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb). If the reviewer has in mind specific parts of the Appendix to be moved to the main paper, please let us know so that we can also address them. \n\nWe would appreciate it if the reviewer could confirm if we have successfully addressed their main comments and thus could consider upgrading the score of our paper accordingly. Please refer to the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb) for a summary of the main contributions of our work.\n\n---\n\n[1] [Indeterminacy in Generative Models: Characterization and Strong Identifiability](https://proceedings.mlr.press/v206/xi23a.html)\n\n[2] [Causal Autoregressive Flows](http://proceedings.mlr.press/v130/khemakhem21a.html)\n\n[3] [Neural Spline Flows](https://proceedings.neurips.cc/paper/2019/hash/7ac71d433f282034e088473244df8c02-Abstract.html)\n"}, {"review_id": "Y6efTqmR3i", "reviewer": "Reviewer_yYim", "summary": "This paper proposes causal normalizing flows, a method to learn a structural causal model (SCM) using normalizing flows (NFs), when the causal order (and potentially more information about the graph) is available. They consider a general SCM where we have observed variables X which follow the equations X_i = f_i(X_pa(i), u_i) where pa(i) are causal parents of i and u_i is endogenous noise. Then, by unrolling the recurisve definition, the authors propose to model this distribution via NFs. In fact, because of acyclicity, the NFs are actually Triangular Monotonic Increasing (TMI) maps. Under the following assumptions\n- The functions f_i are diffeomorphic\n- the causal graph is acyclic\n- the causal ordering is known (this can be a strong assumption, see below)\n- causal sufficiency, i.e. the endogenous variables u_i are mutually independent\nit follows froms prior works that the model is identifiable (means there is a unique product distribution and TMI maps given the observational dataset). The authors restate this result and use it as a basis for their experimental explorations.\n\nThe main selling point of the work is that with approproate design choices of causal normalizing flows, the causal task of do-operations can be performed efficiently. Different design choices for causal normalizing flows are proposed, including generative (modeling the mixing directly) and abductive (modeling the inverse map) methods. To handle do-operations, the authors propose modifying the endogenous variable directly (since the recursive SCM form is lost). They also show how to adapt their framework when the data is discrete or partial knowledge of the graph is available. For experiments, ablation studies compare the design choices, and the framework is also compared to baselines CAREFL and VACA, showing that CNFs perform better on a few SCM tasks. Finally, a fairness use-case is shown with the German credit dataset. The target audience are people interested in causal inference.", "questions": "Please comment on the issues above.", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Both structural causal modeling and normalizing flows (by now) are well-studied topics and this work is a nice interplay between these two frameworks.\n\n- The application to deduce credit risk while being unbiased on sex is an interesting and a bit unusual application of their framework.", "weaknesses": "- Assuming knowledge of causal ordering is usually a very strong requirements in applications, since if we have the knowledge of causal ordering, we can use traditional nonlinear regression techniques, e.g. with spline functions, to learn the underlying SCMs. Therefore, this may potentially be a very limiting assumption in experiments and reduce the usefulness of the proposed method.\n\n- The identifiability results are not novel and therefore the thrust of this work is experimental. As the authors clarify, theorem 1 is taken from Xi and Bloem-Reddy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments, as well as the highly detailed summary of our work. Before commenting on the concerns raised by the reviewer, we would like to clarify a few points that may have been overlooked or misunderstood based on the review’s summary:\n\n> [Identifiable] means there is a unique product distribution and TMI maps given the observational dataset.\n\nThis definition would rather correspond to strong identifiability, which only occurs if the base distributions of the flow and SCM are the same. Otherwise, we obtain weak identifability, where the mapping is not unique, but differs on a component-wise transformation (Theorem 1), as depicted in Figure 3, which is sufficient to enable causal inference.  \n\n> … showing that CNFs perform better on a few SCM tasks.\n\nBased on the “few” above, we are unsure whether this passed unnoticed, but we would like to remind that in Appendix D.4 we provide results for 12 different SCMs. We restricted this number to 3 in the main manuscript due to space constraints.\n\nNext, regarding the reviewer’s concerns:\n\n> Assuming knowledge of causal ordering is usually a very strong requirements in applications, since if we have the knowledge of causal ordering, we can use traditional nonlinear regression techniques, e.g. with spline functions, to learn the underlying SCMs.\n\nWe would like to highlight that we focus in our paper in causal inference, i.e., recovering the value of the exogenous variables to then obtain  estimates for interventional and counterfactual queries. For such tasks, and in contrast to causal discovery problems, it is indeed common to assume access to the causal graph in addition to observational data [[1](https://aaai.org/papers/08159-vaca-designing-variational-graph-autoencoders-for-causal-queries/), [2](http://arxiv.org/abs/2302.00860), [3](https://openreview.net/forum?id=vouQcZS8KfW), [4](https://proceedings.mlr.press/v177/sanchez22a.html), [5](http://arxiv.org/abs/2109.04173)], when no other assumptions are considered (e.g., access to interventional data [[6]](https://proceedings.mlr.press/v202/nasr-esfahany23a.html) or to auxiliary variables [[7]](https://proceedings.mlr.press/v108/khemakhem20a.html)).  Importantly, to the best of our understanding, traditional nonlinear regression techniques for causal inference iteratively  fit a  regressor (e.g., spline function) per node, conditioned on its parents. As a consequence, in addition to assume a known causal graph, parameters are not amortized and errors tend to propagate along causal paths (from root nodes to leave nodes). \n\nYet, as we acknowledge that access to the full causal graph may be restrictive in real-world applications, we provide an algorithm to group variables together if only partial knowledge on the causal ordering is available (the description in lines 171-178 are expanded in Appendix A.2.2).\n\n\n\n\n> The identifability results are not novel and therefore the thrust of this work is experimental.\n\nWe respectfully and strongly disagree with this comment by the reviewer. While we do adapt the theory from Xi and Bloem-Reddy [[8]](https://proceedings.mlr.press/v206/xi23a.html) to the problem of causal inference, we would like to stress that their work involves identifiability of latent variable models in an ICA framework, where there is **not any causal consideration.**  Our main contribution lies in the fact that by casting SCMs and ANFs as TMI maps, we can re-formulate a causal inference problem (i.e., recovering the true exogenous variables)  as a blind-source separation problem, and thus leverage their identifiability results. To the best of our knowledge, our work is the first to provide identifiability guarantees for such a broad family of SCMs (lines 110-113). For ease of exposition, we restated Proposition 5.2 from [[8]](https://proceedings.mlr.press/v206/xi23a.html) to match our setting, which may lead to confusions. We will make this more clear in the next revision of the manuscript.\n\nBesides the  aforementioned identifiability result, the characterization of causally-consistent ANF network designs, an efficient implementation of the do-operator suitable for causal NFs, as well as extensions to mixed-type data and partial causal graphs are also theoretical and novel contributions of our work. We make these points more precise in the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb).\n\n\nWe hope to have addressed the main concerns, as well as clarified potential misunderstandings, raised by the reviewer, and thus would  appreciate it if the reviewer could consider upgrading the score of our paper accordingly. Please refer to the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb) for a summary of the main contributions of our work, as well as for a summary of the changes planned for our paper. \n\n---\n\n[1] [VACA: Design of Variational Graph Autoencoders for Interventional and Counterfactual Queries](https://aaai.org/papers/08159-vaca-designing-variational-graph-autoencoders-for-causal-queries/)\n\n[2] [Interventional and Counterfactual Inference with Diffusion Models](http://arxiv.org/abs/2302.00860)\n\n[3] [Neural Causal Models for Counterfactual Identification and Estimation](https://openreview.net/forum?id=vouQcZS8KfW)\n\n[4] [Diffusion Causal Models for Counterfactual Estimation](https://proceedings.mlr.press/v177/sanchez22a.html)\n\n[5] [Relating Graph Neural Networks to Structural Causal Models](http://arxiv.org/abs/2109.04173)\n\n[6] [Counterfactual Identifiability of Bijective Causal Models](https://proceedings.mlr.press/v202/nasr-esfahany23a.html)\n\n[7] [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://proceedings.mlr.press/v108/khemakhem20a.html)\n\n[8] [Indeterminacy in Generative Models: Characterization and Strong Identifiability](https://proceedings.mlr.press/v206/xi23a.html)\n"}, {"review_id": "J4TLxfObRY", "reviewer": "Reviewer_azJz", "summary": "n this paper, the authors derive and demonstrate the usefulness of normalizing flows (NF) in causal inference. As a first building block of their approach, the authors show that structural causal models (SCMs) can be expressed as triangular monotonic increasing maps.  By this reduction, they show that SCMs can be approximated by autoregressive NFs and that those are causally consistent.\nThey proceed by discussing different architectures to best model the SCM with NFs and conclude that an abductive setup for the NF might be the most suitable architecture.\nIn order to perform interventions on an NF they define the do-operator on those. Due to the recursive representation of their approach, the authors resort to defining an intervention by fixing the exogenous variables accordingly. In empirical evaluations, they show that their approach compares favourably to the SOTA on representing the true data distribution and causal inference tasks w.r.t. to both performance and time efficiency.\nLastly, they show on real-world data that their approach can be used for fairer, but still precise classification.", "questions": "No questions", "limitations": "The authors address the limitations of the work clearly and describe how their approach relies on their assumptions. Furthermore, they describe how violating each of their assumptions would affect the overall approach. These observations seem to be complete w.r.t. the possible limitations of this approach.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The elaboration of the connections between SCMs, TMIs, and ANFs is well executed and can find additional applications in other research.\n- The implementation of interventions as performing them on the exogenous variables is non-trivial and opens up further paths of investigation.\n- Figures 2-4 are very valuable for presentation and understanding.\n- Clear and sound experimental setup.\n- Only mild and standard assumptions.\n- Showcasing the usefulness of causal NFs in ML w.r.t. fairness.", "weaknesses": "- The paper would benefit of an example of how a learned causal NF behaves, how the interventions influence the output, and how it matches an SCM.\n- Consider pushing some of the information from the appendix to the main text. E.g.:\n\t- 152: A sketch of the proof would strengthen the line of argumentation of this work.\n\t- 327: At least a short sentence on what this data is about.\n- Some background on TMI maps could be introduced to make it easier to follow the content.\n- No resources such as codebase, trained models, and analysis implementation are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback, detailed summary, and positive comments. Please see our comments regarding the raised weaknesses below:\n\n> The paper would benefit of an example of how a learned causal NF behaves, how the interventions influence the output, and how it matches an SCM.\n\nWe appreciate the comment, and we would appreciate more specific details on said example to properly address it during the rebuttal. To our understanding, the reviewer may be referring to something similar to the pair-plots we provide in Figures 12, 13, and 15 from the appendix, but we are happy to add other examples.\n\n> Consider pushing some of the information from the appendix to the main text. E.g.: …\n\nWe thank the reviewer for the specific feedback, and we will implement these changes in the updated manuscript.\n\n> Some background on TMI maps could be introduced.\n\nWe agree with the reviewer that more background on TMI maps can be helpful to the reader, and we will add it to the next revision of our work.\n\n> No resources such as codebase, trained models, and analysis implementation are provided.\n\nWe would like to point out that we provide the code to reproduce the experiments of our work with the submission, in the same zip file where the appendix was found. We did not consider necessary providing the model weights, given how quick they are to train and run in all our experiments. In addition, we will make all the code necessary to reproduce all our experiments publicly available with the publication of our paper. \n\nAs for the “analysis implementation”, we are unsure what the reviewer refers to exactly. Could the reviewer clarify, please? We could happily address any additional comments they may have. \n\nIn case we have successfully clarified the points raised by the reviewer, we would appreciate it if  they could consider upgrading the score of our paper accordingly. Please refer to the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb) for a summary of the main contributions of our work.\n"}, {"review_id": "mhcZpfx4cN", "reviewer": "Reviewer_p91A", "summary": "The authors propose a normalizing flow (NF) model that incorporates causal information, either via a partial order or complete DAG. They derive identifiability conditions for causal estimands and implement their algorithm on a range of simulated and real-world datasets. The resulting NF can be used to compute treatment effects and counterfactual probabilities, under some reasonable assumptions on the data generating process. ", "questions": "See above.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a strong contribution on an important topic. The manuscript is well-motivated and thoroughly researched. The writing is clear and the experimental validation is convincing. Despite a few minor questions/comments (see below), I am generally supportive of this work.", "weaknesses": "There are a few minor points that left me somewhat confused. I suspect these could be cleared up with a few brief lines. \n\n-I’m unsure how to interpret $\\sum_{n=1}^\\infty \\mathbf{G}^n$. Why the infinite sum? The examples in Fig. 4 all appear finite? \n\n\n-It appears that the empirical evaluation only considers causal chains of the form $X_1 \\rightarrow \\dots \\rightarrow X_d$. What about more complicated/realistic causal structures? \n\n\n-If I understand Fig. 5 correctly, the ordering model (green curves) outperforms the graph model (orange curves) by KL-divergence and causal effect estimation in generative models when $L < 3$, while the ordering model dominates effect estimation across all numbers of layers in the abductive model. This echoes the results summarized in Table 1. I feel I must be missing something, however – why should a model do better with a causal ordering than it does with the entire graph? Surely the latter is strictly more informative? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and for the supportive words. Please see our comments to the reviewer’s comments below:\n\n> I’m unsure how to interpret $\\sum_{n=1}^\\infty G^n$ . Why the infinite sum? The examples in Fig. 4 all appear finite?\n\nThe short and honest answer is that the infinite is there to save space. Since $G$ is acyclic, that sum is always finite as $G^n$ becomes $0$ for $n > \\operatorname{diam}(G)$, but we did not need to define any notation for the diameter of the graph anywhere else. However, we agree that it can be confusing to the reader, and we will rewrite it as a finite sum with $\\operatorname{diam}(G)$ terms in the next revision of the manuscript.\n\n> It appears that the empirical evaluation only considers causal chains. What about more complicated/realistic causal structures?\n\nWe only consider causal chains for the ablation study shown in the main paper, as using causal chains is easy to understand and sufficient to verify the claims we make in Section 4 with respect to the network design. Yet, we do consider more complex SCMs (in terms of causal graphs and structural equations) in Section 6 and Appendix D.4 (see Figure 9 for a reference on the considered graphs). Moreover,   the use-case in Section 7 shows a more realistic scenario with mixed-type data, and a partial causal graph (see Figure 14).\n\n> If I understand Fig. 5 correctly […] This echoes the results summarized in Table 1: Why should a model do better with a causal ordering than it does with the entire graph? Surely the latter is strictly more informative?\n\nIndeed, Figure 5 echoes Table 1. The goal of the ablation study in Section 6.1 is to empirically corroborate the theoretical findings summarized in Table 1 (e.g., that the generative model with the graph needs as many layers as the graph diameter), as well as to motivate the necessity of a network that is causally consistent by design. Since our theory only requires a causal ordering (Section 3), one could reasonably believe that any ANF would suffice, yet we empirically show that in practice we often fall on local optima with spurious correlations, and that a causally-consistent design effectively restricts the search space, easing the optimization process and, ultimately, yielding more accurate estimates of causal inference estimands.\n\nGiven the positive assessment of our paper by the reviewer and, in case we have successfully clarified the questions above, we would appreciate it if  they could consider upgrading the score to better reflect the contributions of our work. Please refer to the [general comment](https://openreview.net/forum?id=QIFoCI7ca1&noteId=3m1mexttyb) for a summary of said contributions."}]

Author rebuttal:
We thank all the reviewers for their useful comments, which will ultimately help us improve the manuscript. We also want to thank them for their reassuring words towards our work. To name just a few, reviewers acknowledged our effort towards good writing and readability:
> I thoroughly enjoyed reading this paper - Reviewer 515Z

>  Figures 2-4 are very valuable for presentation and understanding - Reviewer azJz

the interdisciplinary character of our work:

> This work is a nice interplay between these two frameworks - Reviewer yYim

as well as its novelty and importance:

> This is a strong contribution on an important topic - Reviewer p91A

Additionally, to help during the rebuttal, we would like this general response to serve as a summary of the contributions of our work, as well as a summary of changes to the next updated manuscript.

**List of contributions**

The contributions of our work can be summarized as follows:

1. We formalize the connection between SCMs and ANFs by rewriting them as members of the same family of data-generating processes (i.e., TMI maps with factorized distributions).

2. Under this family, we rephrase causal inference problems as a specific instance of ICA, for which we can adapt the identifiability results from Xi and Bloem-Reddy [[2]](https://proceedings.mlr.press/v206/xi23a.html).

3. We demonstrate in Corollary 2 that, under the conditions of Theorem 1, being causally consistent is a necessary condition at the global optima when learning causal NFs. We bring such theoretical result into practice by discussing in Section 4 (and empirically validating it in Section 6.1) how to design and learn causally-consistent NFs.

4. We provide a new implementation of the do-operator that is well-suited for any SCM representation beyond its usual recursive formulation, enabling causal inference for the proposed causal NFs. We demonstrate theoretically that this is a proper implementation of the do-operator in App. C, and empirically in App. D.4.

5. We extend the above results to the cases of mixed-type data (App. A.2.1) and partial knowledge on the causal graph/ordering (App. A.2.2) to account for more realistic scenarios and thus make causal NFs applicable in a wide range of domains.

6. Finally, we empirically validate all our findings in 12 datasets (see Section 6 and Appendix D), and demonstrate its potential outreach with a fairness use-case in Section 7.

**Relation to previous work**

First, we would like to highlight that we indeed bring the identifiability results on ICA from Xi and Bloem-Reddy [[2]](https://proceedings.mlr.press/v206/xi23a.html) into the causal setting. We consider this contribution novel and significant as the original paper does not have any relationship to causality but only considers latent variable models.

Second, while CAREFL [[1]](http://proceedings.mlr.press/v130/khemakhem21a.html) focuses on causal discovery, it has hinted a connection between ANFs and SCMs. However, we are the first to: i) properly formalize and generalize (to a broader class of SCMs) this connection using TMI mappings; ii) provide identifiability results for causal NFs; and iii) propose a causal NF network design that is causally-consistent and, hence, can accurately and efficiently learn the underlying SCM generating the data.


**List of changes**

Following the reviewers' suggestions, we will carry out the following changes:

1. We will more clearly explain the intuition behind Figure 1. See response to reviewer 1x9i.
2. We will clarify that Theorem 1 has been re-stated from Xi and Bloem-Reddy [[2]](https://proceedings.mlr.press/v206/xi23a.html) to match our particular setting and ease exposition.
3. We will provide further intuition on the advantage of an abductive network vs. a generative one. In short, causal dependencies from $x$ to $u$ (parents) are sparser than those from $u$ to $x$ (ancestors). See response to reviewer 515Z.
4. We will revise the appendix and push important conclusions back to the manuscript. Of course, we are open to suggestions during the rebuttal. As suggested by reviewer azJz, we will push the following information to the main paper:
    1. A sketch of the proof for Corollary 2 to strengthen the line of argumentation of our work.
    2. A few lines introducing the German Credit dataset in more detail to the reader.
    3. Background on TMI maps. If this text becomes too lengthy, we will write a separate section in the appendix.
5. To improve readability, we will substitute the infinite sums $\sum_{n=1}^\infty G^n$ by the equivalent $\sum_{n=1}^{\operatorname{diam}(G)} G^n$. See response to reviewer p91A.
6. We will apply external feedback and fix typos and small errors. E.g.:
    1. In line 113 we call $u$ the endogenous variables, instead of exogenous.
    2. In the linear examples, we use the letter $G$ for the causal adjacency matrix and the actual linear operation, which can lead to confusions.

---

[1] [Causal Autoregressive Flows](http://proceedings.mlr.press/v130/khemakhem21a.html)

[2] [Indeterminacy in Generative Models: Characterization and Strong Identifiability](https://proceedings.mlr.press/v206/xi23a.html)


Meta-review: This work explores the use of normalizing flows in causal inference. Assuming that data has been generated from a causal model and we know the causal graph as well as the causal ordering, the paper presents causal NFs and shows how to implement the do-operator in causal normalzing flows. This is a timely topic, and all reviewers suggest to accept the paper. I fulls support this.
Decision: Accept (oral)