Causal Reasoning in the Presence of Latent Confounders via Neural ADMG Learning
Authors: Matthew Ashman
Chao Ma
Agrin Hilmkil
Joel Jennings
Cheng Zhang
Venue: ICLR 2023 poster (2023)
Keywords: causality
causal discovery
causal inference
structural equation model
latent confounders
variational inference

Abstract: Latent confounding has been a long-standing obstacle for causal reasoning from observational data. One popular approach is to model the data using acyclic directed mixed graphs (ADMGs), which describe ancestral relations between variables using directed and bidirected edges. However, existing methods using ADMGs are based on either linear functional assumptions or a discrete search that is complicated to use and lacks computational tractability for large datasets. In this work, we further extend the existing body of work and develop a novel gradient-based approach to learning an ADMG with nonlinear functional relations from observational data. We first show that the presence of latent confounding is identifiable under the assumptions of bow-free ADMGs with nonlinear additive noise models. With this insight, we propose a novel neural causal model based on autoregressive flows. This not only enables us to model complex causal relationships behind the data, but also estimate their functional relationships (hence treatment effects) simultaneously. We further validate our approach via experiments on both synthetic and real-world datasets, and demonstrate the competitive performance against relevant baselines.

CAUSAL REASONING IN THE PRESENCE OF LATENT CONFOUNDERS VIA NEURAL ADMG LEARNING
Matthew Ashman Chao Ma"; Agrin Hilmkil', Joel Jennings', Cheng Zhang'
Microsoft Research Cambridge University of Cambridge
mca390cam. ac. uk (chaoma, agr inh 1mkil, joel jennings, cheeiiggs,,,chhhhaaaaa betezha@@microooft..oo
ABSTRACT
Latent confounding has been long-standing standing obstacle for causal reasoning from observational data. One popular approach is to model the data using acyclic di- rected mixed graphs (ADMGs), which describe ancestral relations between vari- ables using directed and bidirected edges. However, existing methods using AD- MGs are based on either linear functional assumptions or a discrete search that is complicated to use and lacks computational tractability for large datasets. In this work, we further extend the existing body of work and develop a novel gradient- based approach learning an ADMG with non-linear functional relations from observational data. We first show that the presence of latent confounding is iden- tifiable under the assumptions of bow-free ADMGs with non-linear additive noise models. With this insight, we propose novel neural causal model based on au- toregressive flows for ADMG learning. This not only enables us determine complex causal structural relationships behind the data in the presence of latent confounding, but also estimate their functional relationships (hence treatment ef- fects) simultaneouslly We further validate our approach via experiments both synthetic and real-world datasets, and demonstrate the competitive performance against relevant baselines.
INTRODUCTION
Learning causal relationships and estimating treatment effects from observational studies is a funda- mental problem in causal machine learning, and has important applications in many areas of social and natural sciences (Pearl, 2010; Spirtes, 2010). They enable us to answer questions in causal na- ture; for example, what the effect on the expected lifespan of patient if increase the dose of X drug? However, many existing methods of causal discovery and inference overwhelmingly rely on the assumption that all necessary information is available. This assumption is often untenable in practice. Indeed, an important, yet often overlooked, form of causal relationships is that of latent confounding; that is, when two variables have an unobserved common cause (Verma & Pearl, 1990). If not properly accounted for, the presence of latent confounding can lead to incorrect evaluation of causal quantities of interest (Pearl, 2009).
Traditional causal discovery methods that account for the presence of latent confoundings, such as the fast causal inference algorithm (FCI) (Spirtes et al., 2000) and its extensions (Colombo et al., 2012; Claassen et al., 2013; Chen al., 2021), rely uncovering equivalence class of acyclic directed mixed graphs (ADMGs) that share the same conditional independencies. Without additional assumptions, however, these methods might return uninformative results as they cannot distinguish between members of the same Markov equivalence class (Bellot & van der Schaar, 2021). More recently, causal discovery methods based on structural causal models (SCMs) (Pearl, 1998) have been developed for latent confounding (Nowzohour et al., 2017; Wang & Drton, 2020; Maeda & Shimizu, 2020; 2021; Bhattacharya et al., 2021). By assuming that the causal effects follow specific functional forms, they have the advantage of being able to distinguish between members of the same Markov equivalence class (Glymour et al., 2019). Yet, existing approaches either rely on restrictive linear functional assumptions (Bhattacharya et al., 2021; Maeda & Shimizu, 2020; Bellot & van der Schaar, 2021), and/or discrete search over the discrete space of causal graphs (Maeda &
Equul contribution. This work done when Matthew Ashman was an intern at Microsoft Research.

---

Shimizu, 2021) that are computationally burdensome and unintuitive to use. As result, modeling non-linear causal relationships between variables in the presence of latent confounders in scalable way remains an outstanding task.
In this work, we seek to utilize recent advances in diffcrentiable causal discovery (Zheng et al., 2018; Bhattacharya et al., 2021) and neural causal models (Lachapelle et al., 2019; Morales-Alvaree et al., 2022; Geffner et al., 2022) to overcome these limitations. Our core contribution to extend the framework of differentiable ADMG discovery for linear models (Bhattacharya et al., 2021) 1) to non-linear cases using neural causal models. This enables us to build scalable and flexible meth- ods capable of discovering non-linear, potentially confounded relationships between variables and perform subsequent causal inference. Specifically, our contributions include:
1. Sufticient conditions for ADMG identifiability with non-linear SCMs (Section 4). We assume: i) the functional relationship follows non-linear additive noise SCM; ii) the effect of observed and latent variables do not modulate each other, and iii) all latent variables con- found pair of non-adjacent observed nodes. Under these assumptions, the underlying ground truth ADMG causal graph is identifiable. This serves foundation for designing ADMG identification algorithms for flexible, non-linear SCMs based on deep generative models.
2. A novel gradient- framework for learning ADMGs from observational data (Sec- tion 5). Based on our theoretical results, we further propose Neural ADMG Learning (N- ADMG), a neural uutoregressivv--low flow-based model capable of learning complex non-linear causal relationships with latent confounding. N-ADMG utilizes variational inference to ap- proximate posteriors over causal graphs and latent variables, whilst simultaneously learning the model parameters via gradient-based optimization. This is more efficient and accurate than discrete search methods, allowing us replace task-specific search procedures with general purpose optimizers.
3. Empirical evaluation on synthetic and real-world datasets (Section 6). We evaluate N- ADMG on a variety of synthetic and real-world datasets, comparing performance with a num- ber of existing causal discovery and inference algorithms. We find that N-ADMG provides competitive or state-of-the-ar results on a range of causal reasoning tasks.
RELATED WORK
Causal discovery with latent confounding. Constraint-based causal discovery methods in the presence of latent confounding have been well-studied (Spirtes et al., 2000; Zhang, 2008; Colombo et al., 2012; Claassen al., 2013; Chen et al., 202 Without further assumptions, these approaches can only identify a Markov equivalence class of causal structures (Spirtes et al., 2000). When certain assumptions are made the data generating process in the form of SCMs (Pearl, 1998), additional constraints can help identify the true causal structure. In the most general case, additional non- parametric constraints have been identified (Verma & Pearl, 1990; Shpitser al., 2014; Evans, 2016). Further refinement can be made through the assumption of stricter SCMs. For example, in the linear Gaussian additive noise model (ANM) case, Nowzohour et (2017) proposes a score- based approach for finding an equivalent class of bow-free acyclic path diagrams. Both Maeda & Shimizu (2020) and Wang Drton (2020) develop Independence tests based approach for linear non-Gaussian ANM case, with Maeda & Shimizu (2021) extending this to more general cases.
Differentiable characterization of causal discovery. All aforementioned approaches employ a search over discrete space of causal structures, which often requires task-specific search proce- dures, and imposes a computational burden for large-scale problems. More recently, (Zheng et al., 2018) proposed differentiable constraint on directed acyclic graphs (DAG), and frames the graph structure learning problem as a differentiable constrained optimization task in the absence of latent confounders. This is further generalized to the latent confounding case (Bhattacharya et al., 2021) through differentiable algebraic constraints that characterize the space of ADMGs. Nonetheless. this work is limited in that it only considers linear Gaussian ANMs.
2

---

3 BACKGROUND
3.1 STRUCTURAL CAUSAL MODELS IN THE ABSENCE OF LATENT CONFOUNDINGS
Unlike constraint- approaches such the PC algorithm, SCMs capture the asymmetry be- tween causal direction through functional assumptions on the data generating process, and have been central to many recent developments in causal discovery (Glymour et al., 2019). Given directed acyclic graph (DAG) G on nodes {1, D), SCMs describe the random variable x (#1, ................................ .p) by 2; ft (xa()))), G2), where C is an exogenous noise variable that is independent of all other variables in the model, pa(i; G) denotes the set of parents of node i in G. and fr describes how x; depends on its parents and the noise Eg. We focus on additive noise SCMs, commonly referred to as additive noise models (ANMs), which take the form
2t fi(Xpo(i G) = fi(xpati())) + G. or fx + in vector form.
(1)
This induces a joint observation distribution py (x (G), where denotes the parameters for functions {ft). Under the additive noise model in Equation 1, the DAG G is identifiable assuming causal minimality and no latent confounders (Peters al., 2014).
3.2 GRAPHICAL REPRESENTATION OF LATENT CONFOUNDERS USING ADMGS
One of the most widely-used graphical representations of causal relationships involving latent con- founding is the so-called acyclic directed mixed graph (ADMGs). ADMGs are an extension of DAGs, that contain both directed edges (-+) and bidirected edges (++) between variables. More concretely, the directed edge z; - 2, indicates that x; is an ancestor of zj. and the bidirected edge I; 2y indicates that x; and x; share common, unobserved ancestor (Richardson & Spirtes, 2002: Tian & Pearl, 2002). An ADMG G over a collection of D variables x = (II,.... .......))))))))... can be described using two binary adjacency matrices: GD DDD for which an entry of I in position (i,j) indicates the presence of the directed edge x; + Ij, and GB E IR DxD for which an entry of 1 in position (i.j) indicates the presence of the bidirected edge 24 ++ Tz. Throughout this paper, we will use the graph notation G to indicate the tuple G {GD, GB}. When using ADMGs to represent latent confounding, causal discovery amounts to learning the matrices GD and GB-
Similar to DAG, an SCM can be specified to describe the causal relationships implied by an ADMG through the so-called magnification process. As formulated in (Peña, 2016), whenever a bidirected edge x; ++ 2, is present according to GB in an ADMG, we will explicitly add a latent node um to represent the latent parent (confounder) of 2; + Zz. Then, the SCM of x; can be written as x by x; (i(pa(x;G)) upa();Gn)) + ei, where upu(s;Gn)) denotes the latent parents of x; in the set of all latent nodes u = (u,,... uMM) added in the magnification process. In compact form, we can write:
xx, u ux) +
The 'magnified SCM' will serve as a practical device for learning ADMGs in this paper. Similar to DAGs, magnified SCMs induce an observational distribution given an ADMG, the magnified SCM is not unique as latent variables may be shared. For example, both {aı < U1 - D2, I2 U2 T3, D3 <- J1 T2, T2 F3, 2z ++ 21 be magnified Therefore, ADMG identifiability does not imply Ug + } and {u, + Iι, un + I2, U1 - Fa).
denoted by pa (x"; G*). Note that
the structural identifiability of the magnified SCM. In this paper, focus ADMG identifiability.
3.3 DEALING WITH GRAPH UNCERTAINTY
Additive noise SCMs only guarantee DAG identifiability in the limit of infinite data. In the finite data regime, there is inherent uncertainty in the causal relationships. The Bayesian approach to causal discovery accounts for this uncertainty in a principled manner using probabilistic modeling (Heckerman et al., 2006), in which one's belief in the true causal graph is updated using Bayes' rule. This approach is grounded in causal decision theory, which states that rational decision-makers maximize the expected utility over a probability distribution representing their belief in causal graphs (Soto et al., 2019). We can model the causal graph G jointly with observations x x²..... ,..., x as
N
(x
G) = p()
II 7=
Pa (x" (G)
(3)
3

---

where 9 denotes the parameters of the likelihood relating the observations the underlying causal graph. In general, the posterior distribution po(Gix'.. ,...,, x") ) is intractable. One approach to circumvent this is variational inference (Jordan et al., 1999; Zhang al., 2018), in which we seek an approximate posterior g (G) that minimises the KI KI-divergence KT. [g (G) [oopoo (G|x :...., This can be achieved through maximization of the evidence lower bound (ELBO), given by
;xxxxx.....)))) .....x")) x-
LELBO
(G) [lo po (x [x]]]]] KL (4s (G)ll
(4)
An additional convenience of this approach is that the ELBO serves as a lower bound to the marginal oog-likelihood pa (x', .x., x²",, ), so it can also be maximized with respect to the model parameters to find the parameters that approximately maximize the likelihood of the data (Geffner et al., 2022).
4 ESTABLISHING ADMGS IDENTIFIABILITY UNDER NON-LINEAR SCMs
To build flexible methods that are capable of discovering causal relationships under the presence of latent confounding, we first need to establish the identifiability of ADMGs under non-linear SCMs. The concept of structural identifiability of ADMGs is formalized in the following definition:
Definition 1 (ADMG structural identifiability). For distribution po(x; G), the ADMG G {Gu: Ge} Y is said to be structurally identifiable from px( G) if there exists no other distribution per (x: G') such that G + G' and px G) (x: G*).
Assuming that our model is correctly specified and Pao (x; GO) denotes the true data generating distribution, then ADMG structural identifiability guarantees that if we find some po(x; G) Poo (x; G9) (by e.g., maximum likelihood learning), we can recover G = Go. In this section, seek to establish sufficient conditions under which the ADMG identifiability is satisfied. Let .D) be a collection of observed random variables, and u = (u....., uM) be a col-
x = lection of unobserved (latent) random variables. Our first assumption is that data generating process can be expressed as a specific non-linear additive noise SCM, in which the effect of the observed and latent variables do not modulate each other, see the functional form below:
Assumption 1. We assume that the data generating process takes the form
[x ul = fcxxx(x;8) 0) + fG 0) + where each element of is independent of all other variables in the model and 0 denotes the param- elers of the non-linear functions Junctiionn
(5)
JGo.x and JGa.u
This assumption is one of the elements that separate from previous work of (Bhattacharya et al., 2021; Maeda & Shimizu, 2020) (linearity is assumed) and (Maeda & Shimizu, 2021) (effects are assumed to be fully decoupled, x4 Em Jim (3 E xpa(t:Go) + Σe Dikk (uk E Up)():Ga)) + <). As discussed in Section 3.2, the discovery of an ADMG between observed variables amounts to the discovery of their ancestral relationships. Therefore, the mapping between ADMGs and their magnified SCMs is not one-to-one, which might cause issues when designing causal discovery methods using SCM-based approaches. To further simplify the underlying latent structure (without losing too much generality), our second assumption assumes that every latent variable is parent- less common cause of pair of cojaadjacent observed variables:
Assumption 2 (Latent variables confound pairs of non-adjacent observed variables). For each latent variable un in the data generating process, there exists a non-adjacent pair I; and x that is unique to 4k, such that r; + U4 + Zzj.
Arguments for Assumption 2 are made by (Pearl & Verma, 1992), who show that this family of causal graphs is very flexible, and can produce the same conditional independencies amongst the observed variables in any given causal graph. Therefore, it has been argued that without loss of generality, we can assume latent variables to be exogenous, and have exactly two non-adjacent children. This assumption also implies that we can specify a magnified SCM to the ADMG as shown in Section 3.2, which allows us to evaluate and optimize the induced likelihood later on.
Meanwhile, Evans (2016) show that this graph family cannot induce all possible observational distribu- tions. In Appendix E we show that in the linear non GauGasssin ANM case there exist certain constraints on marginal distributions that cannot be satisfied under Assumption 2. Nonetheless, we could not derive such constraints for the non-linear case.
4

---

Given Assumptions and 2, now provide three lemmas which allow identify the ADMG matrices, GD and GB. These lemmas extend the previous results in Maeda & Shimizu (2021)
under our new assumptions above. Note that all 9; and 9, functions in the lemmas denote functions satisfying the residual faithfulness condition (Maeda & Shimizu (2021)), as detailed in Appendix A.
Lemma (Case 1). Given Assumptions 1 and 2, then [Gnlk and [G 0 and only if
Vg..gj, [(a; (x ()(3 93 (x-
(6)
Lemma (Case 2). Given Assumptions 1 and 2, then [GBl, = and [Gpkk = if and only if Igs [x: - 9:(x ( ف3 ((,)))
(7)
Lemma (Case 3). Given Assumptions / and 2, then [Galk,] = 0 and [Golk 1 if and only if
4gaa 9jr [(x; 9, (x (( 93 (x,1)
(8)
Eg.. 9j. [[<; 9: (xxx)) (x-x(x))))
(9)
Since each case leads to mutually exclusive conditional independencce/eepeddenc constraints, for any pa (x; G) specified under the assumptions above, there cannot exist some per (x; G²) such that G & G' and (x; G) Po (x; G"), thus structural identifiability is satisfied. This gives our ADMG identifiability result:
Proposition 1 (Identifiability of ADMGs under non-linear SCMs). Assume Assumptions I to hold. Then, GD and GB are identifiable for the data generating process specified in Equation 5.
Whilst theoretically we could test for these conditions stated Lemmas 1 to directly, a more efficient approach is to use differentiable maximum likelihood learning. Assuming the model is cor- rectly specified, then ADMG identifiability ensures the maximum likelihood estimate recovers the true graph in the limit of infinite data (Section 5.5). Hence, we can design the ADMG identification algorithms via maximum likelihood learning.
NEURAL ADMG LEARNING
Whilst we have outlined sufficient conditions under which the underlying ADMG causal structure can be identified, this does not directly provide framework through which causal discovery and inference can be performed. this section, we seek to formulate practical framework for gradient- based ADMG identification. Three challenges that remain are: 1. How
parameterize the magnified SCM models for ADMG to enable learning flexible causal relationships? 2. How can optimise our model in the space of ADMG graphs assumed in Section 4? 3. How do
leam the ADMG causal structure efficiently, whilst accounting for the missing data (u) and graph uncertainties in the finite data regime? In this section, we present Neural ADMG Learning (N- ADMG), a novel framework that addresses all three challenges.
5.1 NEURAL AUTO-REGRESSIVE FLOW PARAMETERIZATION
We assume that our model used learn ADMGs from data is correctly specified. That is, it can be written in the same magnified SCM form in Equation 5:
xx, u] T fGoxx(x;;0 0) + fu e.
(10)
Following Khemakhem et al. (2020), we factorise the likelihood pe px(",, (²) induced by Equa- tion 10 in an autoregressive manner. We can rearrange Equation 10 as
6 = <cv-Sco.xx
(u 0) = 96 (v; 0)
(v;
(11)
where v = (x, u) E DDM and Ō is the magnified adjacency matrix on defined as G... (0. 1) and only if 04 + Vy. This allows us to express the likelihood as
D+M = II
Go =
Pe, (ea("")))))
(12)
5

---

Note that have omitted the (Mooij et al., 2011). Following Geffner et al. (2022), we adopt an efficient, flexible parameterization for the functions fi taking the form (consistent with Assumption 1)
term it is equal to since Õ is acyelic
M ( UyEu
[(()) E1,6
EGness(n) + Ea,i 6
(13)
where f1,s E2,; and & (i 1, MM are MLPs. A naive implementation would require training 3(D+ M) neural networks. Instead, construct these MLPs that their weights shared across nodes as (1,s()) = (ei,) )), (-):()) = (e;; and 4()) = f(e;; :), with e, E DRD+M trainable embedding that identifies the output and input nodes respectively.
5.2 ADMG LEARNING VIA MAXIMIZING EVIDENCE LOWER BOUND
Our ADMG identifiability theory in Section 4 suggests that the ground truth ADMG graph can, in principle, be recovered via maximum likelihood learning of po (x()). However, the aforementioned challenges remain: given a finite number of observations x ,x....,''' x how do we deal with the corresponding missing data u²,... u'...., uu while learning the ADMG? How do we account for graph uncertainties and ambiguities in the finite data regime? To address these issues, N-ADMG takes a Bayesian approach toward ADMG learning. Similar to Section 3.3, we may jointly model the distribution over the ADMG causal graph G, the observations x x!..,.,X,, and the corresponding latent variables u² ,..., N as
Po (x
G) = P()) II pa 1" (G)
(14)
where Po(x", G)) the neural SCM model specified in Section 5.1, denotes the corre- sponding model parameters, and p(G) is some prior distribution over the graph. Our goal is to learn both the model parameters and an approximation to the posterior 4o (o' u G) R po(u', ,...,, N G|x! ..... x This can be achieved jointly using the variational inference frame- work (Zhang et al., 2018; Kingma & Welling, 2013), in which we maximize the evidence lower bound (ELBO) CELBO Φ) < En log (x"")(x") given by
LE.BO(O. Φ) Bgg(C)
(a [logpo(x" ju, G3]
KL (G)
(15)
- (o)
Σ ===1
KL lu (onuu("",,)))) G))]lpo(u*(())
In the following sections, we describe our choice of the graph prior p(G), and approximate pos- terior (o(u), ....., u G) = II, q0 x", G) (G). We will also demonstrate that maximizing CELBO σ) recovers the true ADMG causal graph in the limit of infinite data. See Section 5.5.
5.3 CHOICE OF PRIOR OVER ADMG GRAPHS
As discussed in Section 3.2, the ADMG G can be parameterized by two binary adjacency matri- ces, GD whose entries indicate the presence of directed edge, and G3 whose edges indicate the presence of bidirected edge. As discussed in Section 4, a necessary assumption for structural iden- tifiability is that each latent variable is parent-less confounder of a pair of non-adjacent observed variables. This further implies that the underlying ADMG must be bow-free (both directed and a bidirected edge cannot exist between the same pair of observed variables). This constraint can be imposed by leveraging the bow-free constrain penalty introduced by Bhattacharya et al. (2021),
h(GD. GB) trace (c D + sum (GD o GB)
(16)
which is non-negative and zero only if (GD, is bow-free ADMG. As suggested in Geffner al. (2022), we implement the prior as P()) & exp ( (~>e1 I l where the coefficients and o increased whilst maximizing ( Φ). following an aug- mented Lagrangian scheme (Nemirovski, 1999). Prior knowledge about the sparseness of the graph introduced by penalizing the norms ||G and ||G Il? with scaling coefficients X,1 and X
AeellGBll Salff-ph(GGp.CCC (GGD.GB) ah(GG, GahGG..G)))
(17)

---

5.4 CHOICE OF VARIATIONAL APPROXIMATION
We seek to approximate the intractable true posterior pa pa'u", u _ x*) using the vari- ational distribution (o(u",... ,..,, u G). We assume the following factorized approximate posterior
N go (o ",,G) G) = (G) II go (u((x") 7=1
(18)
For (G).. we use product of Bernoulli distributions for each potential directed edge in GD and bidirected edge GB. For GD. edge existence and edge orientation are parameterized separately using the ENCO parameterization (Lippe et al., 2021). For 90 (x")x")) we apply amortized VI as in VAE literature (Kingma & Welling, 2013), where 4s te(u(x")) is parameterized as a Gaussian distribution whose mean and variance is determined by passing the x" through an inference MLP.
5.5 MAXIMIZING (0 6) RECOVERS THE GROUND TRUTH ADMG
In Section 4, we have proved the structural identifiability of ADMGs. In this section, we further show that under certain assumptions, maximizing (ELBO Φ) recovers the true ADMG graph (de- noted by GO) in the infinite data limit. This result is stated in the following proposition:
Proposition 2 (Maximizing (f Φ) recovers the ground truth ADMG). Assume that:
Assumptions and 2 (hence the identifiability of ADMGs) holds for the model po(x; G).
The model is correctly specified (36" such that pa- (x; GO) recovers the Lata-generating process).
Regularity condition: for all 0 and G we have Ep(x:; log po (x; G)Il < cc.
The variational family of Qo(uuxe x. G) is flexible enough, i.e., it contains Pe (u|xx G)-
Then, the solution (0', d'g (G)) that maximizes (O +) satisfies (G) 8(G = GO).
The proof of Proposition 2 can be found in Appendix B, which justifies performing causal discovery by maximizing ELBO of the N- N-ADMG model. Once the model has been trained and the ADMG has been recovered,
the N-ADMG perform causal inference detailed in Appendix C.
EXPERIMENTS
We evaluate N-ADMG in performing both causal discovery and causal inference number of synthetic and real-worid datasets. Note that we run our model both with and without the bow-free constraint, identified as N-BF-ADMG (our full model) and N-ADMG (for ablation purpose), respec- tively. We compare the performance of our model against five baselines: DECI (Geffner et al., 2022) (which we refer to as N-DAG for consistency), FCI (Spirtes al., 2000), RCD (Maeda & Shimizu, 2020), CAM-UV (Maeda & Shimizu, 2021), and DCD (Bhattacharya et al., 2021). We evaluate the causal discovery performance using Fl scores for directed and and bidirected adjacency. The ex- pected values of these metrics are reported using the learned graph posterior (which is deterministic for RCD, CAM-UV, and DCD). Causal inference is evaluated using the expected ATE as described in Appendix C. We evaluate the causal inference performance of the causal discovery benchmarks by fixing (G)) to either deterministic or uniform categorical distributions on the learned causal graphs, then learning non-linear flow-based ANM by optimizing Equation 15 in identical manner to N-ADMG. A full list of results and details of the experimental set-up are included in Appendix F. Our implementation will be available https: thub. cwt..ccco/mmmmmmmmmmommooe. wtw.cooooooccccooooooooc ctafff//cciiiiio
6.1 SYNTHETIC
DATASET
We construct a synthetic fork-collider dataset consisting of five nodes (Figure la). The data- generating process a non-linear ANM with Gaussian noise. Variable pairs (Iz, 2z), and (æy, xa) latent-confounded, whereas (æ1, share observed confounder. We evaluate both causal dis- covery and inference performances. For discovery, we evaluate F-score measure on both Go and GB. For causal inference, choose 244 the treatment variable taking and F2, T3 and 25 as the
7

---

21
SI
FBB
23
24
33
a4
(a) Ground truth.
(h) N-BF-ADMG.
(c) N-ADMG (no constraint).
Y2
F1
I2
F2
25
(d) RCD.
(e) DCD.
(i) CAM-UV.
Figure 1: ADMG identification results on fork-collider dataset.
response variables, and evaluate ATE RMSE to benchmark performance. It is therefore crucial that discovery methods do misidentify latent confounded variables having a direct cause between them, as this would result in biased ATE estimates.
Figure 1b-1f visualizes the causal dis- covery results of different methods, and Table summarizes both discovery and inference metrics. While no meth- ods can fully recover the ground truth graph, method N-BF-ADMG-G and CAM-UV achieved the best overall per- formance. N-BF-ADMG-G and N-
Table 1: Causal discovery and inference results for the fork-collider dataset. The table shows the mean and stan- dard error results across five different random seeds.
METHOD
GD FSCORE
GB FSCORE ATE RMSE
N-BF-ADMG-G
0.64 (0.06)
(0.07)
0.022 (0.003)
N-ADMG-G
0.49 (0.02)
0.99 (0.00)
0.239 (0.067)
N-DAG-G
0.50 (0.00)
0.00 (0.00)
0.046 (0.025)
ADMG-G on average are able to recover all bidirected edges from the data, while CAM-UV can only recover half of the latent variables. Without the bow-free constraint, N-ADMG-G discovers di- rected edge from 24 to F3, which results poor ATE RMSE performance. On other hand, DAG-based method (N-DAG-G) is not able to deal with latent confounders, resulting in the poor f-scores in both GD and Gp. Linear ANM-based methods (RCD and DCD) perform significantly worse than other methods, resulting in f-scores for directed matrices and largest ATE errors. This demonstrates the necessity for introducing non-linear assumptions.
FCI
0.00 (0.00)
0.75 (0.00)
0,072 (0.015)
RCD
0.00 (0.00)
0.54 (0.00)
0.206 (0.029)
CAM-UV
0.80 (0.00)
0.67 (0.00)
0.017 (0.003)
DCD
0.00 (0.00)
0.67 (0.00) 0.208 (0.064)
6.2 RANDOM CONFOUNDED ER SYNTHETIC DATASET
We generate synthetic datasets from ADMG extension of Erdös-Rényí (ER) graph model (Lachapelle et al., 2019). We first sample random ADMGs from ER model, and simulate each variable using a randomly sampled nonlinear ANM. Latent confounders are then removed from the training See Appendix G.2 for details. We consider the number of nodes, directed edges and latent confounders triplets (d, m) {(4, 6, 2), (8, 20, 6), (12, 50, 10)}. The resulting datasets are identified as ER(d, e, m). Figure 2 compares the performance of N-ADMG with the baselines. All variants of N-ADMG outperform the basclines for most datasets, highlighting its effectiveness rel- ative to other methods (even those that employ similar assumptions). Similar to the fork-collide dataset, we see that methods operating under the assumption of linearity perform poorly when the Gata-generating process is non-linear. is worth noting that even when the exogenous noise of ADMG is misspecified, its still exceeds that of other methods in most cases. This robustness is desirable property as in many settings the form of exogenous noise unknown.
6.3 INFANT HEALTH AND DEVELOPMENT PROGRAM (IHDP) DATASET
For the real-world datasets, we evaluate treatment effect estimation performances on infant health and development program data (IHDP). This dataset contains measurements of both infants and their mother during real-life data collected a randomized experiment. The main task to estimate the effect of home visits by specialists on future cognitive test scores of infants, where the ground
8

---

(4.6..4)
BR
ER112 50.
Go adjarency Facore
1.0
Go orisntaionn Faoore
0.8
0.8
0.88
0.4
0.4
0.
02
0.0
0.
HADMG-G
N-DMS59
war 4DM0-G
Ner NOAolo
N-DAG- Pry
RDS
CRM
DSD
HADMS-G
NADME
N-BF A-M--G
N NOAAll
A7885
NDAS Ria
RoD
CAM-UN UV
DoD
RADMM--
N-AD8G-9
Nar ADMG-G
N-DAG
Rai
RCD
CSM
DCD
Figure Causal discovery results for synthetic datasets. For readability, the N-ADMG results are connected with lines. The figure shows mean results across five randomly generated datasets.
truth outcomes are simulated as in (Hill, 2011). To make the task more challenging, additional con- foundings are introduced by removing a subset (non-white mothers) of the treated population. More details can be found in Appendix G.3. We first perform causal discovery to learn the underlying ADMG of the dataset, and then perform causal inference. Since the true causal graph is unknown, we evaluate the causal inference performance of each method by estimating the ATE RMSE.
Apart from the aforementioned baselines, here we introduce four methods: PC-DWL (PC algo- rithm for discovery, DoWhy (Sharma et al., 2021) linear adjustment for inference); PC-DwNL (PC for discovery, DoWhy double machine learning for inference); N-DAG-G-DwL (N-DAG Gaussian for discovery, linear adjustment for inference); and N-DAG-S-DwL (N-DAG Spline for discovery, linear adjustment for inference). Results are summarized in Figures 3a to 3c. Generally, models with non-Gaussian exogenous assumptions tend to have lower ATE estimation errors; while models with linear assumptions (RCD and DCD) have the worst ATE RMSE. Interestingly, the DoWhy-baser plug-in estimators tend to worsen the performances of SCM models. However, regardless of the assumptions made on exogenous noise, our method (N-BF-AI) N--BF-DDG-G and N-BF-ADMG-S consis- tently outperforms all other baselines with the same noise. It is evident that for causal inference in real-world datasets, the ability of N-BF-ADMG to handle latent confoundings and nonlinear causal relationships becomes very effective.
E
ATE RMSE
ATE RMSE
ATE RMSE
0.15
0.15
0.10
0.10
0.05
NSF ADMG-G
R-DAG-G
FCI-G
RCD-G
CAM-U--
DCDGG
M-BF ADMG-S
N-DA0.9
FDX-s
RCD-9
CAMANS
DCD
NOAGGG Det.
N-DAG-5 Die
PC Det
PC Detit
(a) Gaussian exogenous noise.
(b) Spline exogenous noise.
(c) DoWhy baselines.
Figure 3: Causal inference results for the IHDP dataset. The figure shows mean + standard error
results across five random initialisations.
7 CONCLUSION AND FUTURE WORK
In this work, we proposed Neural ADMG Learning (N-ADMG), a novel framework for gradient- based causal reasoning in the presence of latent confounding for nonlinear SCMs. We established identifiability theory for nonlinear ADMGs under latent confounding, and proposed a practical ADMG learning algorithm that is both flexible and efficient. In future work, we will further extend our framework on how to more general settings (e.g., the effect from observed and latent variables modulate in certain forms; latent variables confound adjacent variables; etc), and how to im- prove certain modelling choices such as variational approximation qualities over both causal graph and latent variables

---

REPRODUCIBILITY STATEMENT
number of efforts have been made/will be made for the sake of reproducibility. First, we open source our package in our github page gi thub. addition, in this paper we included clear explanations of any assumptions and a complete proof of the claims the Appendix, as well model settings and hyperparameters. All datasets used in this paper are either publicly available data, or synthetic data whose generation process is described in detail.
crrosoff/causicooo
O. O. In
REFERENCES
Alexis Bellot and Mihaela van der Schaar. Deconfounded score method: Scoring dags with dense unobserved confounding. arXiv preprint arXiv:2103.15106, 2021.
Rohit Bhattacharya, Tushar Nagarajan, Daniel Malinsky, and lla Shpitser. Differentiable causal discovery under unmeasured confounding. In International Conference on Artificial Intelligence and Statistics, pp. 2314-2322. PMLR, 2021.
Ruichu Cai, Feng Xie, Clark Glymour, Zhifeng Hao, and Kun Zhang. Triad constraints for learning causal structure of latent variables. Advances neural information processing systems, 32, 2019.
Wenyu Chen, Mathias Drton, and Ali Shojaic. Causal structural learning via local graphs. arXiv preprint 107.020977 2021.
Tom Claassen, Joris M Mooij, and Tom Heskes. Learning sparse causal models is not NP-hard. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 172- 181, 2013.
Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high- dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pp. 294-321, 2012.
Alexander D' Amour. On multi-cause approaches to causal inference with unobserved counfound- ing: Two cautionary failure cases and a promising alternative. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Ar- lificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp- 3478-3486 3486. PMLR, 16-18 Apr 2019.
Robin J Evans. Graphs for margins Bayesian networks. Scandinavian Journal of Statistics, 43 (3):625-)))) 648, 2016.
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.00 02000....99...... 2022.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graph- ical models. Frontiers in genetics, 10:524, 2019.
David Heckerman, Christopher Meek, and Gregory Cooper. A bayesian approach to causal discov- cry. In Innovations in Machine Learning, pp. 1-28. Springer, 2006.
Jennifer Hill. Bayesian nonparametric modelling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183223,, 1999.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ICA: A unifying framework. In International Conference Artificial In- telligence and Statistics, pp. 2207-2217. PMLR, 2020.
Diederik Kingma and Max Welling. 131.v:1312.6111 2013.
Auto-encoding variational Bayes. arXiv preprint
10

---

Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-baad based neural DAG learning. arXiv preprint arxiv:1906.02226, 2019.
Phillip Lippe. Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclic- ity constraints. arXiv preprint arXiv:2107.10483, 2021.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information pro- cessing systems, 30, 2017.
Takashi Nicholas Maeda and Shohei Shimizu. RCD:
Repetitive causal discovery linear non- gaussian acyclic models with latent confounders. In International Conference Artificial Intel- ligence and Statistics, pp. 735-745. PMLR, 2020.
Takashi Nicholas Maeda and Shohei Shimizu. Causal additive models with unobserved variables. In Uncertainty Artificial Intelligence, pp. 97-106. PMLR, 2021.
Sara Mohammad- Taheri, Robert Ness, Jeremy Zucker, and Olga Vitek. Do-calculus enables causal reasoning with latent variable models. arXiv preprint arXiv:2102.0663: 2021.
Joris M Mooij, Dominik Janzing, Tom Heskes, and Bernhard Schölkopf. On causal discovery with cyclic additive noise model. Advances in neural information processing systems 24, 2011.
Pablo Morales-Alvarez, Wenbo Gong, Angus Lamb, Simon Woodhead, Simon Peyton Jones, Nick Pawlowski, Miltiadis Allamanis, and Cheng Zhang. Simultaneous missing value imputation and structure learning with groups. In Advances in Neural Information Processing Systems, 2022.
Nemirovski. Optimization II. Numerical methods for nonlinear continuous optimization. 1999.
Christopher Nowzohour, Marloes H Maathuis, Robin J Evans, and Peter Bühlmann. Distributional equivalence and structure learning for bow-free acyclic path diagrams. Electronic Journal of Statistics, 102:53342-5374, 2017.
Judea Pearl. Graphs, causality, and structural equation models. Sociological Methods & Research, 27(2):226-284, 1998.
Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, 3:96-146, 2009.
Judea Pearl. An introduction to causal inference. The International Journal of Biostatistics, 6(2): 1-62, 2010.
Judea Pearl. On measurement bias in causal inference. arXiv preprint arXiv:1203.3504 2012.
Judea Pearl and Thomas S Verma. A statistical semantics for causation. Statistics and Computing, 2(2):91-95 1992.
Jose M Peña. Learning acyelic directed mixed graphs from observations and interventions. In Conference on Probabilistic Graphical Models, pp. 392 402. PMLR, 2016.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. 2014.
Rajesh Ranganath and Adler Perotte. Multiple causal inference with latent confounding. arXiv preprint arXiv:1805.08277 2018.
Thomas Richardson and Peter Spirtes. Ancestral graph Markov models. The Annals of Statistics, 30 (4):962-1030, 2002.
Amit Sharma, Vasilis Syrgkanis, Cheng Zhang, and Emre Kıeıaan Dowhy: Addressing challenges in expressing and validating causal assumptions. arXiv preprint arXiv:2108. 13518, 2021.
Ilya Shpitser, Robin J Evans, Thomas Richardson, and James M Robins. Introduction to nested markov models. Behaviormetrika, 41(1):3-39, 2014.

---

Mauricio Gonzalez Soto, David Danks, Hugo Escalante Balderas, and L Enrique Sucar. Choos- ing with unknown causal information: Action-outcome probabilities for decision making can be grounded in causal models. arXiv preprint arXiv:1907.11733 2019.
Peter Spirtes. Introduction causal inference. Journal of Machine Learning Research, 11(5), 2010.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Jin Tian and Judea Pearl. A general identification condition for causal effects. In Aaaifiaai, pp- 567-573, 2002.
Dustin Tran and David M Blei. Implicit causal models for genome-wide association studies. arXiv preprint arXiv: 10742, 2017.
Thomas Verma and Judea Pearl. Equivalence and synthesis of causal models. In Proceedings of the Sixth Annual Conference on Uncertainty in Artificial Intelligence, pp. 255-270, 1990.
Y Samuel Wang and Mathias Drton. Causal discovery with unobserved confounding and non- Gaussian data. arXiv preprint 1311, 2020.
Yixin Wang and David M Blei. The blessings of multiple causes. Journal of American Statistical Association, 114652):157--1596 2019.
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, (0)))2000202266 2018.
Jiji Zhang. On the completeness orientation rules for causal discovery in presence of latent confounders and selection bias. Artificial Intelligence,
2008
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with tears: Continu- ous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse non- parametric DAGs. In International Conference on Artificial Intelligence and Statistics, pp. 3414- 3425. PMLR, 2020.
12

---

APPENDIX
A PROOF OF LEMMAS TO 3
First we describe the residual faithfulness condition inherited from (Maeda & Shimizu, 2021) in Lemmas to 3:
Definition 2 (Residual faithfulness condition). We say that nonlinear functions gis 9j satisfies the residual faithfulness condition, if: for any two arbitrary subset of x. denote by M and N, when both (I:- 9o ()) and (x+-9; (M)) have terms involving the same exogenous noise Ck2 then (et-):(M)) and (z; Oj (M) are mutually dependent.
Next, we provide the proof for Lemmas help of the following lemma, which extends Lemma A in (Maeda & Shimizu, 2021), except we don't assume any form for Is other than non-linearity.
Section 4. To prove those lemmas, we need the
Lemma 4. Let s(xx) denote an arbitrary function of zz. The residual of s(x,) regressed onto x-; cannot be independent ef:
Vgt, [s(x;) 9:(x Kea).
(19)
Proof. Assume that [s(x;)- -x-xx:) holds, then x-; must contain at least one descendent of aa as it must have dependence on the noise 6, to cancel effect of & s(x;). We can express 9: 9xxx;) as (E(.). Since De operates on variables defined by non-linear transformations of the exogenous noise terms, we cannot express u; as a:(6_;) :)+ + by (e()). x-; contains a descendent of 7i, so 6-; includes at least noise term éş that satisfies lee (i.e. is not in æ.). Thus, terms containing e, cannot be fully removed from s(x;) - (x(x and so [s(x,) - g (x (x) [e does not hold.
A.1 PROOF OF LEMMA 1
Proof. Define yi and gj (x(x fax (pa (0))+ and g;((x)) lz (pax respectively. Then, Equation 6 becomes equivalent to
Vgl. dy. (pa,,( (i)) + ((U (i)) + + -x
(20)
Given Lemma 4 and following the same arguments in Maeda & Shimizu (2021), this is equivalent to
(fi,lpa, (1)) ())
+g).
(21)
Since e,, we have
(fa, (e,) (niKKf,,upa, (f. (P).
(22)
The first implies the existence of an unobserved mediator between and 24, the second implics the existence of unobserved mediator between and 2,, and the third implies the existence of an unobserved confounder. Given the assumption of latent variables being confounders and minimality. this indicates the presence of latent confounder and direct cause between o; and 2z--
A.2 PROOF OF LEMMA 2
Proof. When Equation 7 holds, Equation does not. Thus, there is no unobserved confounder between I; and zy. Assume that zj is a direct cause I₁, and that Equation is satisfied for ga
and 9j. 2; contains a nonlinear function of e, that cannot be removed by (x(xx(())) (i.s)). thus (z,
gi(x (..)) Similarly, (z, 9 (x (i.))) Thus, we have [[x; (x(x) M(x, gj (x(,))))))) )] which contradicts our initial assumption. The same arguments apply when is
(i.))
direct cause of zy, implying that there can be no causal relationship between n, and xy.
A.3 PROOF OF LEMMA 3
Proof. When Equation holds, Equation 6 does not and so there is no latent confounder between x; and Zj. When Equation& 8 holds, Equation does not hold and so there is a direct causal relationship between z; and 2.--
13

---

Assume that 2 is direct cause of 2;- Define 91 (x(x)) = (ix((ax and -2(xx(is)) (i,))
Jix (pax(i)), giving gi (x-+) (pa + and 2j ()(x)((s)) (pa (j)) +
ey. When there is no latent confounder, fi,u (pa, (i)) (1) (pa
Thus, (f.u(pa, (i) +
(f), (i)) Cj) hods and Equation satisfied.
Now, assume that z; is direct cause of z... Using Lemma 4, we have
Vgi, [[2, gi (x
(23)
Similarly, since zy is a function of x, we also have
Agg. [xx 9) (x (i.5) Kcsl.
(24)
Collectively, this implies
Vgs 9j. [x; - ( g)(x
(25)
which contradicts Equation Thus, if Equation satisfied then zÿ a direct cause of a;-
B PROOF OF PROPOSITION 2
To prove Proposition we need the following lemma:
Lemma 5. Assume variational distribution (G) over a space of graphs G6, where each graph G E G has non-zero associated weight WGG With the soft prior p(G) defined as Equation 17 and bounded X1, X2, P. a, we have
lim - (G)l(G 0.
(26)
N N
Proof. This directly follows from Lemma 1 of (Geffner et al., 2022).
L
Now we can proceed to prove Proposition
Proof.
For N-ADMG, in the infinite data limit LELBO becomes
lim No0
E (G)atu llogrexx, un(G)) - N
Σ
Hiau Xn. G)]
KL (c()))l(())
N
n=1
n=1
0
lim
wp(G)),(ulx.c) [log (x Un, G)]
Σ
(G) [KL (q x., G)) Gp),p Xx,, G)]]
N
GEG
(27)
where the zeroing of the KL divergence follows from Lemma 5. Given fixed 6, the optimal posterior q" (u,, X,, G) satisfies q" (u,, xx, G) = (u, G) due to the flexibility assumption. Thus,
lim (O, Φ, (un/xn, G))
N N
lim N
Σ
ws(G)log (x,
(28)
GEGs
= =f p(x; GO)
Σ
wo(G) log pe (x (xxx
GES
where pxx; GO) denotes the data generation distribution with the ground truth ADMG, G Let (8", G²) = arg max J p(x; log (x G)dx be the MLE solution. Since Egeg, ws(G) 1,
(G) 0, we have
Σ GEG.
ue (G) [xo])((((G]] S Ep(e;99) [log po- (x; G*)
14

---

with the optimal value of Egeg, (o(G))p(x;co) plog is achieved when every graph G Go and associated parameter fg satisfies
Bp(x;69) Peg (x|G) = p [log Pa Pg
(x]
(29)
Since the model is correctly specified, the MLE solution (0", G*) satisfies
Ep(x,oy) [log pa- (x|G")] = E p(x
p(x; GO)
Therefore, condition Equation 29 implies for every graph G' E Gb. G' GD under the regularity condition; or equivalently. Gø = {G' =G This proves our statement that d'g (G) = 8 = G'), where G' G
C ESTIMATING TREATMENT EFFECTS
For all experiments we consider, the causal quantity of interest we wish to estimate is the expected average treatment effeet (ATE), (G) (ATE(a, b[G)], where the expectation is taken with respect to our learned posterior over causal graphs (G))):
(G) [ATB(a, a, b[G) = Egg(C) [Epcxy =b) [xy] Bp(xy|do(xx== [xy]]-
(30)
This requires samples from p(xy |do(x; b), G) = p(xy |XT = b, Gdo(x))): ) where Gan(ez) is the 'mutilated' graph obtained by removing incoming edges into Xy. We can achieve this by simulating the learnt SCM on Gdo(xr) whilst keeping XT = b fixed. Note that Te (u|x) is not used to estimate the ATE; it suffices to sample u from the prior distribution, p(u). In our setting, the inference MLP qo(ut (u|x) is only used as means through which the likelihood of the data can be evaluated efficiently, and thus model parameters learned. This is in a similar spirit to VAEs (Kingma & Welling, 2013).
D RELATED WORK ON CAUSAL INFERENCE UNDER LATENT CONFOUNDING
Attempting to perform causal inference in the presence of latent confounding can lead to biased esti- mates (Pearl, 2012). Whilst the observed data distribution may still be identifiable, estimating causal effects are not (Spirtes et al., 2000). A recent string of work has made progress in the case where the effects of multiple interventions 2018; Wang & Blei, 2019; D' Amour, 2019). An alternative approach is to assume identifiability of the joint distribution over both latent and observed variables given just the observations. Louizos et al. (2017) point out that there are many cases in which this is possible (Khemakhem et al., 2020; Kingma & Welling, 2013). Nevertheless, all these methods assume the underlying causal graph is known. More recently, Mohammad-1 Taheri et al. (2021) argue that a DAG latent variable model trained on data can be used for down-stream causal inference tasks even if its parameters are non- identifiable, long as the query can be identified from the observed variables according to the do-calculus.
being estimated (Tran & Blei, 2017: Ranganath & Perotte,
E ADDITIONAL DISCUSSIONS ON STRUCTURAL IDENTIFIABILITY OF LATENT VARIABLES
In Section 3.2, we argued that the identifiability ADMG does not imply the structural identifiabil- ity of the magnified SCM. In this section, we will present more discussions on certain identifiability of latent structures (of magnified SCMs). In general, these examples demonstrate that for linear non-Gaune non-Gaussian ANMs the structure of latent variables can be refined beyond the assumption of latent confounders acting between pairs of non con-adjacent observed variables, whilst the same techniques cannot achieve the same for non-linear ANMs
15

---

E.1 DETERMINING CAUSAL STRUCTURE AMONGST LATENT VARIABLES
Recently, Cai et al. (2019) demonstrated that it is possible to discover the structure amongst latent variables using their so-called Triad constraints. Their method limited the linear non aon-aassian ANM case. In this section, we demonstrate that an analogous constraint isn is''ten available for non- non-linear ANMs.
f12
f21
f
f2j
f2k
fus
f2 Y.
f2k
T2
Zzj
Ik
24
IJ
It
(a)
(b)
Figure 4: The two possible latent variable structures considered by Cai et al. (2019).
Consider the causal graphs shown in Figure 4a and Figure 4b.
Lemma 6 (Linear non- non-Gause Gaussian identifiabilityy. In the linear non- non-Gaussian ANM case, Equation 31 satisfied only for the causal graph shown Figure 4b. the non-linear ANM case, Equation is not sutisfied for either Figure 4a Figure 4b.
3g.
g(z,) L
(31)
Sketch of Proof. For the causal graph in Figure 4a, Equation 31 is equivalent to
3g f(((1)) + ((12)(1221)) e,) 122 u2) + ex.
(32)
On the right, we have some function of the latent variables and uz, fex (212f12(1 (11) u2). On the left, we have some function of the same noise terms, fi [u,) both sides, we require y to be non-zero and so term including u2 is still present. To remove u2, again require; y to be non-zero and so term including uı still present. Thus, Equation is not satisfied in either the linear non Gaussian non con-linear ANM case.
(f12((())))) Gj). To remove in from
For the causal graph in Figure 4b, Equation 31 is equivalent to
3g f1s(J21 (u2), u1) - e; g(22((u2) + (2) + +e.-
(33)
In the linear case, we can construct linear g to remove U2 fom the left side so that Equation holds (i.e. g fx.f2x), ). In the non-linear case, U1 and U2 are coupled in the leftmost term and cannot be removed by a term involving u2 and ej. Hence, Equation 31 violated.
fas
E.2 DETERMINING THE NUMBER OF LATENT CONFOUNDERS
Here, we consider whether the number of latent confounders can determined in the non- non-linear ANM case. Lemma shows that in the linear non-Gaussian case, the number of latent confounders acting between a triplet of confounded observed variables can be determined (by verifying certain constraints marginal distributions observed variables), whilst the same approach cannot be used in the non-linear ANM case.
Consider the two causal graphs shown in Figure 5a and Figure 5b.
Lemma (Linear non- Gaussaan identifiability.. In the linear non- non-Gaussian ANM case. Equation 34 satisfied only for the causal graph shown Figure 4b. In the non- non-linear ANM case, Equation 34 not satisfied for either Figure 4a Figure 4b.
3g, 2; g(z;)
(34)
Sketch of Proof. For Figure 5b, Equation 34 is equivalent
Eg S((u) I(5(u)) (e) + ek.
(35)
16

---

In the linear non- non-Gausiian case, it is straightforward to get to remove the common noise term from the left term and make the two sides independent. In the non-linear case, when fi * fj the common noise term cannot be removed from the left as 9 must be non-linear, and thus produces term that involves both u and Cj. For Figure 5a, Equation is equivalent to
3g fu u2) + e; 9 (u,, u3) + Il f (u2, U3) + Ek.
(36)
U2 cannot be removed from the left, and so Equation 34 does not hold in either the linear non- Gaussian non-linear case.
u1
U2
U3
fui
3
2k
f3k
fk
j
Jo
Ijj
Ok
(a)
(b)
Figure 5: Two possible latent structures that confound each pair of variables 2,, 2 and zk.
F OPTIMISATION DETAILS
F.1 OPTIMISATION DETAILS FOR N-ADMG
As discussed in Section 5, we gradually increase the prior hyperparameters and a throughout training. This is done using the augmented Lagrangian procedure for optimisation (). The optimisa- tion process interleaves two steps: 1) optimise the objective for fixed values of p and a for a certain number of steps; and ii) update the values of p and a. Steps and ii) and ran until convergence, or the maximum number of optimisation steps is reached. We describe these two steps in more detail below.
Step i). The objective is optimised for some fixed vales of p and a using Adam O. We use learning rate of 10-3 for the model parameters and 5 x 10-3 for the variational parameters. We optimise the objective for maxmimum of 5000 steps until convergence (we stop early if the loss does not improve for 1500 optimisation steps, moving step ii)). During training, we reduce the learning rate by factor of 10 if the training loss does not improve for 1000 steps a maximum of two times. If we reach the condition a third time, we assume optimisation has converged and move to step ii). We apply annealing to the KL-divergence between the approximate posterior and prior over the latent variables. The annealing contant fixed for each step i), and increased linearly over the first optimisation loops.
Step ii). We initialise p = and a = 0. At the beginning of step i) we measure the DAG / bow-free penalty Pt = (G) [h[G)]. At the beginning of step ii), measure this penalty again, P2 Eggicy [A]G If P2 < P1, leave p unchanged and update + + pP2. Otherwise, if P2 2 0.65P1, we leave a unchanged and update p + 10p. We repeat the steps i) to ii) a maximum p reaching some value which set 10
of 30 times until convergence (measured for both).
F.2 ADDITIONAL HYPERPARAMETERS
Prior Hyperparameters.
We use the sparsity inducing prior hyperparameters As,1 Ax
We construct an approximation to the ELBO in Equation 15 using a single sample from the approximate posteriors. For evaluating the gradients of the ELBO use the Gumbel softmax method with a hard forward pass and soft backward pass with temperature of 0.25.
ELBO approximation.
Neural network architectures. The functions & E and é used in the likelihood and the inference network used to parameterise 9 (u|x) are all two hidden layer MLPs with 80 hidden units per hidden layer.
17

---

Non-Gaussian noise model. For the non-Gaussian noise model
ATE estimation. For ATE estimation we compute expectations by drawing 1000 graphs from the learnt posterior, and for each graph we draw samples of Xy for a total of 2000 samples which we used to form a Monte Carlo estimate.
G DATASET DETAILS
G.1 SYNTHETIC FORK-COLLIDE DATASET
We constructed a 2000 sample synthetic dataset with the causal structure shown in Figure 1 by sampling from the following SEM:
W1, U2 G1s C Cs ca. cs| (0.I)
22 VGexp(-uf + 0.1e2
(37)
43 = V6exp(-uf + --expp(-aa (-12) + 0.2e3
24 Voexp(-15 (5)) + V6exp(-at + 0.1c4
Z5 = VGexp(-af) + 0.
Variables in and U2 are latent confounders acting on variable pairs v2 and Vg. and ag and r4, respectively.
G.2 LATENT CONFOUNDED ER DATASET
We generate synthetic datasets from ADMG extension of Erdös-Rényi (ER) graph model (Lachapelle et al., 2019; Zheng et al., 2020). An ER(d, e, dataset is generated according the following procedures:
1. Generate d x directed adjacency matrix Gp of ADMG from Endös-Rényí (ER) graph model, whose expected directed edges equal to e;
2. Simulate d d bidirected adjacency matrix GB via random Bernoulli sampling, whose expected number equals to m;;
Simulate exogenous noises e, from a zero mean Gaussian distribution with standard deviation of
0.1;
4. Simulate latent variables from zero mean Gaussian distribution with standard deviation of 0.1;
Simulate each observed variable x; f((Xpa(;GG)) + 9s (upa((G))) + Gs where fú, D: are randomly sampled nonlinear functions of the form: y w c -x
6. Remove u from the sampled dataset.
G.3 IHDP DATASET DETAILS
This dataset contains measurements of both infants (birth weight, head circumference, etc.) and their mother (smoked cigarettes, drank alcohol, took drugs, etc) during real-life data collected in randomised experiment. The main task is to estimate the effect of home visits by specialists on fu- ture cognitive test scores of infants. The outcomes of treatments are simulated artificially as in Hill (2011); 1); hence the outcomes of both treatments (home visits or not) on each subject are known. Note that for each subject, our models are only exposed to only one the treatments; the outcomes the other potential/coouttrfactuu outcomes are hidden from the mode, and are only used for the purpose of ATE evaluation. To make the task more challenging, additional confoundings are manually intro- duced by removing subset (non-white mothers) of the treated children population. In this way can construct the IHDP dataset of 747 individuals with continuous covariates and 19 binary covari- ates. We 10 replicates of different simulations based setting B (log-linear response surfaces)
18

---

mathod
N-BF-AFMG-G
Run time (minutes)
300
CAM-UV
200
100
2000
4000
6000
10000
Number of datapoints
Figure 6: Run time results for N-BF-ADMG and CAM-UV on a 12 variable synthetic ER dataset. The figure shows mean results standard deviation across five randomly generated datasets.
of Hill (2011), which can downloaded from /tttts://githhii S.tm/A/laab-AAmtterra//EEEV We use 70%/30% train-test split ratio. Before training our models, all continuous covariates are normalised.
H RUN TIME COMPARISON
Here, compare the run time of N- N-BB-ADMG and CAM-UV on synthetically generated ER(12, (12, 50, 10) dataset. The results are shown Figure N-BF-ADMG is trained using 30k epochs to ensure convergence, whose Figure highlight that methods based on continuous optimisation (N-BF-ADMG) offer significant improvement in run time relative methods based conditional independency tests (CAM-UV) for large datasets. In combination with the results in the main paper, this shows the empirical validity of deep learning approach for identifying ADMGs that scales to larger dataset, without sacrificing accuracy.
time be further reduced via early stopping The results in
19


Reviews:
[{"review_id": "Udl7tCdRry", "reviewer": "Reviewer_DadL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a gradient based approach to causal discovery in the presence of unobserved confounding. By making assumptions on the structure of the graph along with additive noise assumptions - identification results for non-linear SCMs are provides. Since the conditional independencies involves learning functions, a neural ADMG learning framework is proposed. Finally, empirical evaluation is performed where the method is shown to perform well. ", "summary_of_the_review": "Overall, this paper presents a novel causal discovery algorithm under the assumption of additive noise. Identification results are first established, then the proposed neural ADMG learning framework is used to learn an ADMG. This paper is novel because it allows for learning ADMGs while relaxing assumptions made by previous works since, it allows for non-linear functions in the SCM. Next, instead of relying on traditional conditional independence tests - which can sometimes run into tractability issues in high dimensions, the neural ADMG framework is proposed that is more suitable for dealing with high dimensional data. High quality paper, I recommend for acceptance. ", "strength_and_weaknesses": "Strengths: This paper establishes causal discovery results on a more flexible class of ADMGs than what exists in the literature. Proper identification conditions are established, and the inference framework has computational tractability advantages.\n\nWeakness: The statistical framework for neural ADMG learning does require a lot of modelling decisions when it comes to things such as choice of variational approximation - and I do worry how how much flexibility these methods lose when such choices are made. However, I understand the necessity of these assumptions, and I consider this only a minor weakness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RX87FoZDzB", "reviewer": "Reviewer_fy3p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method of estimating ADMGs from observational data. The authors build on existing work on ADMG identifiability and on graph learning with deep neural networks. First, two assumptions (sufficient conditions) are proposed that guarantee identifiability of the underlying ADMG (inspired by Maeda and Shimizu, 2021). Subsequently, a variational scheme (N-ADMG) for learning the ADMG is introduced (based on Geffner et al., 2022). The paper concludes with a series of experiments showing causal discovery (ADMG recovery) and causal inference (average treatment effect estimation).", "summary_of_the_review": "This paper combines two existing ideas which leads to a somewhat novel approach to ADMG estimation from observable data.", "strength_and_weaknesses": "STRENGTHS\n\nWhile this paper builds heavily on existing work, both in deriving the sufficient conditions for ADMG identifiability and for specifying the variational model optimized with neural nets, the two parts fit together nicely which leads to a well-defined model for ADMG estimation.\n\nThe research area is also very relevant and has received a lot of attention recently (deep causal models, accounting for unobserved confounding).\n\n\nWEAKNESSES\n\nThe novelty of the proposed approach is somewhat limited by the fact that the vital building blocks (ADMG identification criteria and variational causal graph structure learning) are already present in the literature.\n\nThe experiments appear inconclusive, especially given the fact that both in the causal discovery and causal inference (ATE estimation) experiment, the methods coming close to (or beating) N-ADMG are the ones corresponding to the \"building blocks\", i.e. N-DAG and CAM-UV.\n\nAs a minor issue, Figures 2 and 3 are difficult to read.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L6A6h21Fhi", "reviewer": "Reviewer_ZhRx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors modeled the data structure with acyclic directed mixed graphs (ADMGs). Under the condition where latent confounding is present, a new autoregressive  flow-based approach is proposed to learn the causal and functional relations of the given data. In the experiments. the authors validate competitive performance of their approach in comparison to the baselines.", "summary_of_the_review": "Considering the violation of ICLR template's layout and the length of the paper, I would kindly suggest submitting this paper to a journal instead. ", "strength_and_weaknesses": "Strengths: \n1 I like the mathematical formulation discussed in Sec. 4. The introduction of ADMG identifiability seems to be quite clear. Though the additive noise assumption and confounding assumption are simplified, the authors achieve a good balance between practicality in real-world application and technicality in mathematical reasoning. The discussions from  lemma 1-3 also make sense to me and fairly easy to understand.\n\n2. The autoregressive approach that is implemented with MLP is well-motivated. The authors have justified the method on both  a toy and real datasets.\n\nWeakness:\n1 I think the major issue is that the authors altered the layout of the ICLR template and significantly decrease the space between sections.\nI am not sure if this is fair for other authors who follow the submission guidelines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FcQxupAW0y", "reviewer": "Reviewer_dczA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors generalize the differentiable causal discovery method to the setting in the presence of latent variables. Compared to related work, the method can tackle non-linear data. They provide both theoretical and empirical results.", "summary_of_the_review": "Overall, the quality is good. The authors propose a differentiable method for learning ADMG under the current causal discovery framework, with theoretical guarantees and rational experiments. Hence, I give a positive score. The main reason for the borderline score is that I do not quite agree with the study regarding learning the ADMG (this discussion is possibly beyond the scope of this paper.). The assumptions necessary for ADMG learning are too strong. I am strongly skeptical whether such a target of learning ADMG can be achieved in practice, and whether this study can help the practical problems. ", "strength_and_weaknesses": "Strength:\n1. The paper is clearly written. \n2. The authors give a detailed review of the related studies. \n3. There are some new theoretical results in this paper.\n\nWeaknesses:\nI do not find some significant weaknesses except for some typos. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster