The Causal-Neural Connection: Expressiveness, Learnability, and Inference
Year: 2021

Abstract:
One of the central elements of any causal inference is an object called structural causal model (SCM), which represents a collection of mechanisms and exogenous sources of random variation of the system under investigation (Pearl, 2000). An important property of many kinds of neural networks is universal approximability: the ability to approximate any function to arbitrary precision. Given this property, one may be tempted to surmise that a collection of neural nets is capable of learning any SCM by training on data generated by that SCM. In this paper, we show this is not the case by disentangling the notions of expressivity and learnability. Specifically, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020), which describes the limits of what can be learned from data, still holds for neural models. For instance, an arbitrarily complex and expressive neural net is unable to predict the effects of interventions given observational data alone. Given this result, we introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences. Building on this new class of models, we focus on solving two canonical tasks found in the literature known as causal identification and estimation. Leveraging the neural toolbox, we develop an algorithm that is both sufficient and necessary to determine whether a causal effect can be learned from data (i.e., causal identifiability); it then estimates the effect whenever identifiability holds (causal estimation). Simulations corroborate the proposed approach.

heading: 1 Introduction
text: One of the most celebrated and relied upon results in the science of intelligence is the universality of neural models. More formally, universality says that neural models can approximate any function (e.g., boolean, classification boundaries, continuous valued) with arbitrary precision given enough capacity in terms of the depth and breadth of the network [14, 26, 47, 53]. This result, combined with the observation that most tasks can be abstracted away and modeled as input/output – i.e., as functions – leads to the strongly held belief that under the right conditions, neural networks can solve the most challenging and interesting tasks in AI. This belief is not without merits, and is corroborated by ample evidence of practical successes, including in compelling tasks in computer vision [43], speech recognition [22], and game playing [54]. Given that the universality of neural nets is such a compelling proposition, we investigate this belief in the context of causal reasoning.
To start understanding the causal-neural connection – i.e., the non-trivial and somewhat intricate relationship between these modes of reasoning – two standard objects in causal analysis will be instrumental. First, we evoke a class of generative models known as the Structural Causal Model (SCM, for short) [58, Ch. 7]. In words, an SCMM∗ is a representation of a system that includes a collection of mechanisms and a probability distribution over the exogenous conditions (to be formally defined later on). Second, any fully specified SCMM∗ induces a collection of distributions known as the Pearl Causal Hierarchy (PCH) [5, Def. 9]. The importance of the PCH is that it formally delimits
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
distinct cognitive capabilities (also known as layers; not to be confused with neural nets layers) that can be associated with the human activities of “seeing” (layer 1), “doing” (2), and “imagining” (3) [59, Ch. 1]. 1 Each of these layers can be expressed as a distinct formal language and represents queries that can help to classify different types of inferences [5, Def. 8]. Together, these layers form a strict containment hierarchy [5, Thm. 1]. We illustrate these notions in Fig. 1(a) (left side), where SCMM∗ induces layers L∗1, L∗2, L∗3 of the PCH.
Even though each possible statement within these capabilities has well-defined semantics given the true SCM M∗ [58, Ch. 7], a challenging inferential task arises when one wishes to recover part of the PCH whenM∗ is only partially observed. This situation is typical in the real world aside from some special settings in physics and chemistry where the laws of nature are understood with high precision.
For concreteness, consider the setting where one needs to make a statement about the effect of a new intervention (i.e., about layer 2), but only has observational data from layer 1, which is passively collected.2 Going back to the causalneural connection, one could try to learn a neural model N using the observational dataset (layer
1) generated by the true SCMM∗, as illustrated in Fig. 1(b). Naturally, a basic consistency requirement is that N should be capable of generating the same distributions asM∗; in this case, their layer 1 predictions should match (i.e., L1 = L∗1). Given the universality of neural models, it is not hard to believe that these constraints can be satisfied in the large sample limit. The question arises of whether the learned model N can act as a proxy, having the capability of predicting the effect of interventions that matches the L2 distribution generated by the true (unobserved) SCMM∗. 3 The answer to this question cannot be ascertained in general, as will become evident later on (Corol. 1). The intuitive reason behind this result is that there are multiple neural models that are equally consistent w.r.t. the L1 distribution ofM∗ but generate different L2-distributions. 4 Even though N may be expressive enough to fully representM∗ (as discussed later on), generating one particular parametrization of N consistent with L1 is insufficient to provide any guarantee regarding higher-layer inferences, i.e., about predicting the effects of interventions (L2) or counterfactuals (L3).
The discussion above entails two tasks that have been acknowledged in the literature, namely, causal effect identification and estimation. The first – causal identification – has been extensively studied, and general solutions have been developed, such as Pearl’s celebrated do-calculus [57]. Given the impossibility described above, the ingredient shared across current non-neural solutions is to represent assumptions about the unknownM∗ in the form of causal diagrams [58, 65, 7] or their equivalence classes [28, 60, 29, 71]. The task is then to decide whether there is a unique solution for the causal query based on such assumptions. There are no neural methods today focused on solving this task. The second task – causal estimation – is triggered when effects are determined to be identifiable by the first task. Whenever identifiability is obtained through the backdoor criterion/conditional ignorability [58, Sec. 3.3.1], deep learning techniques can be leveraged to estimate such effects with impressive practical performance [63, 52, 48, 31, 69, 70, 35, 64, 15, 25, 37, 30]. For effects that are identifiable through causal functionals that are not necessarily of the backdoor-form (e.g., frontdoor,
1This structure is named after Judea Pearl and is a central topic in his Book of Why (BoW), where it is also called the “Ladder of Causation” [59]. For a more technical discussion on the PCH, we refer readers to [5].
2The full inferential challenge is, in practice, more general since an agent may be able to perform interventions and obtain samples from a subset of the PCH’s layers, while its goal is to make inferences about some other parts of the layers [7, 46, 5]. This situation is not uncommon in RL settings [66, 17, 44, 45]. Still, for the sake of space and concreteness, we will focus on two canonical and more basic tasks found in the literature.
3We defer a more formal discussion on how neural models could be used to assess the effect of interventions to Sec. 2. Still, this is neither attainable in all universal neural architectures nor trivially implementable.
4Pearl shared a similar observation in the BoW [59, p. 32]: “Without the causal model, we could not go from rung (layer) one to rung (layer) two. This is why deep-learning systems (as long as they use only rung-one data and do not have a causal model) will never be able to answer questions about interventions (...)”.
napkin), other optimization/statistical techniques can be employed that enjoy properties such as double robustness and debiasedness [32, 33, 34]. Each of these approaches optimizes a particular estimand corresponding to one specific target interventional distribution.
Despite all the great progress achieved so far, it is still largely unknown how to perform the tasks of causal identification and estimation in arbitrary settings using neural networks as a generative model, acting as a proxy for the true SCMM∗. It is our goal here to develop a general causal-neural framework that has the potential to scale to real-world, high-dimensional domains while preserving the validity of its inferences, as in traditional symbolic approaches. In the same way that the causal diagram encodes the assumptions necessary for the do-calculus to decide whether a certain query is identifiable, our method encodes the same invariances as an inductive bias while being amenable to gradient-based optimization, allowing us to perform both tasks in an integrated fashion (in a way, addressing Pearl’s concerns alluded to in Footnote 4). Specifically, our contributions are as follows: 1. [Sec. 2] We introduce a special yet simple type of SCM that is amenable to gradient descent called a neural causal model (NCM). We prove basic properties of this class of models, including its universal expressiveness and ability to encode an inductive bias representing certain structural invariances (Thm. 1-3). Notably, we show that despite the NCM’s expressivity, it still abides by the Causal Hierarchy Theorem (Corol. 1). 2. [Sec. 3] We formalize the problem of neural identification (Def. 8) and prove a duality between identification in causal diagrams and in neural causal models (Thm. 4). We introduce an operational way to perform inferences in NCMs (Corol. 2-3) and a sound and complete algorithm to jointly train and decide effect identifiability for an NCM (Alg. 1, Corol. 4). 3. [Sec. 4] Building on these results, we develop a gradient descent algorithm to jointly identify and estimate causal effects (Alg. 2). There are multiple ways of grounding these theoretical results. In Sec. 5, we perform experiments with one possible implementation which support the feasibility of the proposed approach. All appendices including proofs, experimental details, and examples can be found in the full technical report [68].

heading: 1.1 Preliminaries
text: In this section, we provide the necessary background to understand this work, following the presentation in [58]. An uppercase letter X indicates a random variable, and a lowercase letter x indicates its corresponding value; bold uppercase X denotes a set of random variables, and lowercase letter x its corresponding values. We use DX to denote the domain of X and DX = DX1 × · · · × DXk for X = {X1, . . . , Xk}. We denote P (X) as a probability distribution over a set of random variables X and P (X = x) as the probability of X being equal to the value of x under the distribution P (X). For simplicity, we will mostly abbreviate P (X = x) as simply P (x). The basic semantic framework of our analysis rests on structural causal models (SCMs) [58, Ch. 7], which are defined below. Definition 1 (Structural Causal Model (SCM)). An SCMM is a 4-tuple 〈U,V,F , P (U)〉, where U is a set of exogenous variables (or “latents”) that are determined by factors outside the model; V is a set {V1, V2, . . . , Vn} of (endogenous) variables of interest that are determined by other variables in the model – that is, in U ∪V; F is a set of functions {fV1 , fV2 , . . . , fVn} such that each fi is a mapping from (the respective domains of) UVi ∪PaVi to Vi, where UVi ⊆ U, PaVi ⊆ V \ Vi, and the entire set F forms a mapping from U to V. That is, for i = 1, . . . , n, each fi ∈ F is such that vi ← fVi(paVi ,uVi); and P (u) is a probability function defined over the domain of U.
Each SCMM induces a causal diagram G where every Vi ∈ V is a vertex, there is a directed arrow (Vj → Vi) for every Vi ∈ V and Vj ∈ Pa(Vi), and there is a dashed-bidirected arrow (Vj L9999K Vi) for every pair Vi, Vj ∈ V such that UVi and UVj are not independent. For further details on this construction, see [5, Def. 13/16, Thm. 4]. The exogenous UVi’s are not assumed independent (i.e. Markovianity does not hold). We will consider here recursive SCMs, which implies acyclic diagrams, and that the endogenous variables (V) are discrete and have finite domains.
We show next how an SCMM gives values to the PCH’s layers; for details on the semantics, see [5, Sec. 1.2]. Superscripts are omitted when unambiguous. Definition 2 (Layers 1, 2 Valuations). An SCMM induces layer L2(M), a set of distributions over V, one for each intervention x. For each Y ⊆ V,
PM(yx) = ∑
{u|Yx(u)=y}
P (u), (1)
where Yx(u) is the solution for Y after evaluatingFx := {fVi : Vi ∈ V\X}∪{fX ← x : X ∈ X}. The specific distribution P (V), where X is empty, is defined as layer L1(M).
In words, an external intervention forcing a set of variables X to take values x is modeled by replacing the original mechanism fX for each X ∈ X with its corresponding value in x. This operation is represented formally by the do-operator, do(X = x), and graphically as the mutilation procedure. For the definition of the third layer, L3(M), see Def. 9 in Appendix A or [5, Def. 7].

heading: 2 Neural Causal Models and the Causal Hierarchy Theorem
text: In this section, we aim to resolve the tension between expressiveness and learnability (Fig. 1). To that end, we define a special class of SCMs based on neural nets that is amenable to optimization and has the potential to act as a proxy for the true, unobserved SCMM∗. Definition 3 (NCM). A Neural Causal Model (for short, NCM) M̂(θ) over variables V with parameters θ = {θVi : Vi ∈ V} is an SCM 〈Û,V, F̂ , P (Û)〉 such that
• Û ⊆ {ÛC : C ⊆ V}, where each Û is associated with some subset of variables C ⊆ V, and DÛ = [0, 1] for all Û ∈ Û. (Unobserved confounding is present whenever |C| > 1.) • F̂ = {f̂Vi : Vi ∈ V}, where each f̂Vi is a feedforward neural network parameterized by θVi ∈ θ mapping values of UVi ∪ PaVi to values of Vi for some PaVi ⊆ V and UVi = {ÛC : ÛC ∈ Û, Vi ∈ C}; • P (Û) is defined s.t. Û ∼ Unif(0, 1) for each Û ∈ Û.
There is a number of remarks worth making at this point.
1. [Relationship NCM→ SCM] By definition, all NCMs are SCMs, which means NCMs have the capability of generating any distribution associated with the PCH’s layers. 2. [Relationship SCM 9 NCM] On the other hand, not all SCMs are NCMs, since Def. 3 dictates that Û follows uniform distributions in the unit interval and F̂ are feedforward neural networks. 3. [Non-Markovianity] For any two endogenous variables Vi and Vj , it is the case that UVi and UVj might share an input from Û, which will play a critical role in causality, not ruling out a priori the possibility of unobserved confounding and violations of Markovianity. 4. [Universality of Feedforward Nets] Feedforward networks are universal approximators [14, 26] (see also [19]), and any probability distribution can be generated by the uniform one (e.g., see probability integral transform [1]). This suggests that the pair 〈F̂ , P (Û)〉 may be expressive enough for modelingM∗’s mechanisms F and distribution P (U) without loss of generality. 5. [Generalizations / Other Model Classes] The particular modeling choices within the definition above were made for the sake of explanation, and the results discussed here still hold for other, arbitrary classes of functions and probability distributions, as shown in Appendix D.
To compare the expressiveness of NCMs and SCMs, we introduce the following definition. Definition 4 (P(Li)-Consistency). Consider two SCMs, M1 and M2. M2 is said to be P(Li)consistent (for short, Li-consistent) w.r.t.M1 if Li(M1) = Li(M2).
This definition applies to NCMs since they are also SCMs. As shown below, NCMs can not only approximate the collection of functions of the true SCMM∗, but they can perfectly represent all the observational, interventional, and counterfactual distributions. This property is, in fact, special and not enjoyed by many neural models. (For examples and discussion, see Appendix C and D.1.) Theorem 1 (NCM Expressiveness). For any SCMM∗ = 〈U,V,F , P (U)〉, there exists an NCM M̂(θ) = 〈Û,V, F̂ , P (Û)〉 s.t. M̂ is L3-consistent w.r.t.M∗. Thm. 1 ascertains that there is no loss of expressive power using NCMs despite the constraints imposed over its form, i.e., NCMs are as expressive as SCMs. One might be tempted to surmise, therefore, that an NCM can be trained on the observed data and act as a proxy for the true SCM M∗, and inferences about other quantities ofM∗ can be done through computation directly in M̂. Unfortunately, this is almost never the case: 5
5Multiple examples of this phenomenon are discussed in Appendix C.1 and [5, Sec. 1.2]
Corollary 1 (Neural Causal Hierarchy Theorem (N-CHT)). Let Ω∗ and Ω be the sets of all SCMs and NCMs, respectively. We say that Layer j of the causal hierarchy for NCMs collapses to Layer i (i < j) relative toM∗ ∈ Ω∗ if Li(M∗) = Li(M̂) implies that Lj(M∗) = Lj(M̂) for all M̂ ∈ Ω. Then, with respect to the Lebesgue measure over (a suitable encoding of L3-equivalence classes of) SCMs, the subset in which Layer j of NCMs collapses to Layer i has measure zero.
This corollary highlights the fundamental challenge of performing inferences across the PCH layers even when the target object (NCM M̂) is a suitable surrogate for the underlying SCMM∗, in terms of expressiveness and capability of generating the same observed distribution. That is, expressiveness does not mean that the learned object has the same empirical content as the generating model. For concrete examples of the expressiveness of NCMs and why it is insufficient for causal inference, see Examples 1 and 2 in Appendix C.1. Thus, structural assumptions are necessary to perform causal inferences when using NCMs, despite their expressiveness. We discuss next how to incorporate the necessary assumptions into an NCM to circumvent the limitation highlighted by Corol. 1.

heading: 2.1 A Family of Neural-Interventional Constraints (Inductive Bias)
text: In this section, we investigate constraints about M∗ that will narrow down the hypothesis space and possibly allow for valid cross-layer inferences. One well-studied family of structural constraints comes in the form of a pair comprised of a collection of interventional distributions P and causal diagram G, known as a causal bayesian network (CBN) (Def. 15; see also [5, Thm. 4])). The diagram G encodes constraints over the space of interventional distributions P which are useful to perform cross-layer inferences (for details, see Appendix C.2). For simplicity, we focus on interventional inferences from observational data. To compare the constraints entailed by distinct SCMs, we define the following notion of consistency:
used as an inductive bias for M̂ .
Definition 5 (G-Consistency). Let G be the causal diagram induced by SCMM∗. For any SCMM, we say thatM is G-consistent (w.r.t.M∗) if G is a CBN for L2(M).
In the context of NCMs, this means thatM would impose the same constraints over P as the true SCM M∗ (since G is also a CBN for L2(M∗) by [5, Thm. 4]). Whenever the corresponding diagram G is known, one should only consider NCMs that are G-consistent. 6 We provide below a systematic way of constructing G-consistent NCMs. Definition 6 (C2-Component). For a causal diagram G, a subset C ⊆ V is a complete confounded component (for short, C2-component) if any pair Vi, Vj ∈ C is connected with a bidirected arrow in G and is maximal (i.e. there is no C2-component C′ for which C ⊂ C′.) Definition 7 (G-Constrained NCM (constructive)).
Let G be the causal diagram induced by SCMM∗. Construct NCM M̂ as follows. (1) Choose Û s.t. ÛC ∈ Û if and only if C is a C2-component in G. (2) For each variable Vi ∈ V, choose PaVi ⊆ V s.t. for every Vj ∈ V, Vj ∈ PaVi if and only if there is a directed edge from Vj to Vi in G. Any NCM in this family is said to be G-constrained.
Note that this represents a family of NCMs, not a unique one, since θ (the parameters of the neural networks) are not yet specified by the construction, only the scope of the function and independence relations among the sources of randomness (Û). In contrast to SCMs where both 〈F , P (u)〉 can freely vary, the degrees of freedom within NCMs come from θ. 7
6Otherwise, the causal diagram can be learned through structural learning algorithms from observational data [65, 61] or experimental data [41, 40, 27]. See the next footnote for a neural take on this task.
7There is a growing literature that models SCMs using neural nets as functions, but which differ in nature and scope to our work. Broadly, these works assume Markovianity, which entails strong constraints over P (U)
We show next that an NCM constructed following the procedure dictated by Def. 7 encodes all the constraints of the original causal diagram.
Theorem 2 (NCM G-Consistency). Any G-constrained NCM M̂(θ) is G-consistent.
We show next the implications of imposing the structural constraints embedded in the causal diagram.
Theorem 3 (L2-G Representation). For any SCMM∗ that induces causal diagram G, there exists a G-constrained NCM M̂(θ) = 〈Û,V, F̂ , P (Û)〉 that is L2-consistent w.r.t.M∗.
The importance of this result stems from the fact that despite constraining the space of NCMs to those compatible with G, the resultant family is still expressive enough to represent the entire Layer 2 of the original, unobserved SCMM∗. Fig. 2 provides a mental picture useful to understand the results discussed so far. The true SCMM∗ generates the three layers of the causal hierarchy (left side), but in many settings only observational data (layer 1) is visible. An NCM M̂ trained with this data is capable of perfectly representing L1 (right side). For almost any generatingM∗ sampled from the space Ω∗, there exists an NCM M̂ that exhibits the same behavior with respect to observational data (M̂ is L1-consistent) but exhibits a different behavior with respect to interventional data. In other words, L1 underdetermines L2. (Similarly, L1 and L2 underdetermine L3 [5, Sec. 1.3].) Still, the true SCMM∗ also induces a causal diagram G that encodes constraints over the interventional distributions. If we use this collection of constraints as an inductive bias, imposing G-consistency in the construction of the NCM, M̂ may agree with those of the trueM∗ under some conditions, which we will investigate in the next section.

heading: 3 The Neural Identification Problem
text: We now investigate the feasibility of causal inferences in the class of G-constrained NCMs. 8 The first step is to refine the notion of identification [58, pp. 67] to inferences within this class of models.
Definition 8 (Neural Effect Identification). Consider any arbitrary SCMM∗ and the corresponding causal diagram G and observational distribution P (V). The causal effect P (y | do(x)) is said to be neural-identifiable from the set of G-constrained NCMs Ω(G) and observational distribution P (V) if and only if P M̂1(y | do(x)) = P M̂2(y | do(x)) for every pair of models M̂1, M̂2 ∈ Ω(G) s.t. PM ∗ (V) = P M̂1(V) = P M̂2(V).
In the context of graphical identifiability [58, Def. 3.2.4] and do-calculus, an effect is identifiable if any SCM in Ω∗ compatible with the observed causal diagram and capable of generating the observational distribution matches the interventional query. If we constrain our attention to NCMs, identification in the general class would imply identification in NCMs, naturally, since it needs to hold for all SCMs. On the other hand, it may be insufficient to constrain identification within the NCM class, like in Def. 8, since it is conceivable that the effect could match within the class (perhaps in a not very expressive neural architecture) while there still exists an SCM that generates the same observational distribution and induces the same diagram, but does not agree in the interventional query; see Example 7 in Appendix C. The next result shows that this is never the case with NCMs, and there is no loss of generality when deciding identification through the NCM class.
and, in the context of identification, implies that all effects are always identifiable; see Corol. 3. For instance, [21] attempts to learn the entire SCM from observational (L1) data, while [8, 10] also leverages experimental (L2) data. On the inference side, [42] focuses on estimating causal effects of labels on images.
8This is akin to what happens with the non-neural CHT [5, Thm. 1] and the subsequent use of causal diagrams to encode the necessary inductive bias, and in which the do-calculus allows for cross-layer inferences directly from the graphical representation [5, Sec. 1.4].
Theorem 4 (Graphical-Neural Equivalence (Dual ID)). Let Ω∗ be the set of all SCMs and Ω the set of NCMs. Consider the true SCMM∗ and the corresponding causal diagram G. Let Q = P (y | do(x)) be the query of interest and P (V) the observational distribution. Then, Q is neural identifiable from Ω(G) and P (V) if and only if it is identifiable from G and P (V).
In words, Theorem 4 relates the solution space of these two classes of models, which means that the identification status of a query is preserved across settings. For instance, if an effect is identifiable from the combination of a causal graph G and P (v), it will also be identifiable from G-constrained NCMs (and the other way around). This is encouraging since our goal is to perform inferences directly through neural causal models, within Ω(G), avoiding the symbolic nature of do-calculus computation; the theorem guarantees that this is achievable in principle.
Corollary 2 (Neural Mutilation (Operational ID)). Consider the true SCM M∗ ∈ Ω∗, causal diagram G, the observational distribution P (V), and a target query Q equal to PM∗(y | do(x)). Let M̂ ∈ Ω(G) be a G-constrained NCM that is L1-consistent withM∗. If Q is identifiable from G and P (V), then Q is computable through a mutilation process on a proxy NCM M̂, i.e., for each X ∈ X, replacing the equation fx with a constant x (Q = PROC-MUTILATION(M̂ ; X = x,Y)).
Following the duality stated by Thm. 4, this result provides a practical, operational way of evaluating queries in NCMs: inferences may be carried out through the process of mutilation, which gives semantics to queries in the generating SCMM∗ (via Def. 2). What is interesting here is that the proposition provides conditions under which this process leads to valid inferences, even whenM∗ is unknown, or when the mechanismsF and exogenous distribution P (U) ofM∗ and the corresponding functions and distribution of the proxy NCM M̂ do not match. (For concreteness, refer to example 5 in Appendix. C.) In words, inferences using mutilation on M̂ would work as if they were onM∗ itself, and they would be correct so long as certain stringent properties were satisfied – L1-consistency, G-constraint, and identifiability. As shown earlier, if these properties are not satisfied, inferences within a proxy model will almost never be valid, likely bearing no relationship with the ground truth. (For fully worked out instances of this situation, refer to examples 2, 3, or 4 in Appendix C).
Still, one special class of SCMs in which any interventional distribution is identifiable is called Markovian, where all Ui are assumed independent and affect only one endogenous variable Vi.
Corollary 3 (Markovian Identification). Whenever the G-constrained NCM M̂ is Markovian, P (y | do(x)) is always identifiable through the process of mutilation in the proxy NCM (via Corol. 2).
Algorithm 1: Identifying/estimating queries with NCMs.
Input : causal queryQ = P (y | do(x)), L1 data P (V), and causal diagram G Output : PM ∗ (y | do(x)) if identifiable, FAIL otherwise.
1 M̂ ← NCM(V, G) // from Def. 7 2 θ∗min←argminθ P
M̂(θ)(y |do(x)) s.t. L1(M̂(θ))=P(V) 3 θ∗max←argmaxθ P
M̂(θ)(y |do(x)) s.t. L1(M̂(θ))=P(V) 4 if PM̂(θ ∗ min)(y | do(x)) 6= PM̂(θ ∗ max)(y | do(x)) then 5 return FAIL 6 else 7 return PM̂(θ ∗ min)(y | do(x)) // choose min or max
arbitrarily This is obviously not the case for general nonMarkovian models, which leads to the very problem of identification. In these cases, we need to decide whether the mutilation procedure (Corol. 2) can, in principle, produce the correct answer. We show in Alg. 1 a learning procedure that decides whether a certain effect is identifiable from observational data. Intuitively, the procedure searches for two models that respectively minimize and maximize the target query while maintaining L1-consistency with the data distribution. If the L2 query values induced by the two models are equal, then the effect is identifiable, and the value is returned; otherwise, the effect is non-identifiable. Remarkably, the procedure is both necessary and sufficient, which means that all, and only, identifiable effects are classified as such by our procedure. This implies that, theoretically, deep learning could be as powerful as the do-calculus in deciding identifiability. (For a more nuanced discussion of symbolic versus optimization-based approaches for identification, see Appendix C.4. For non-identifiability examples and further discussion, see C.3.)
Corollary 4 (Soundness and Completeness). Let Ω∗ be the set of all SCMs,M∗ ∈ Ω∗ be the true SCM inducing causal diagram G, Q = P (y | do(x)) be a query of interest, and Q̂ be the result from running Alg. 1 with inputs P ∗(V) = L1(M∗) > 0, G, and Q. Then Q is identifiable from G and P ∗(V) if and only if Q̂ is not FAIL. Moreover, if Q̂ is not FAIL, then Q̂ = PM ∗ (y | do(x)).

heading: 4 The Neural Estimation Problem
text: While identifiability is fully solved by the asymptotic theory discussed so far (i.e., it is both necessary and sufficient), we now consider the problem of estimating causal effects in practice under imperfect optimization and finite samples and computation. For concreteness, we discuss next the discrete case with binary variables, but our construction extends naturally to categorical and continuous variables (see Appendix B). We propose next a construction of a G-constrained NCM M̂(G;θ) = 〈Û,V, F̂ , P (Û)〉, which is a possible instantiation of Def. 7: V := V, Û := {UC : C ∈ C2(G)} ∪ {GVi : Vi ∈ V}, F̂ := { fVi := arg maxj∈{0,1} gj,Vi + { log σ(φVi(paVi ,u c Vi ; θVi)) j = 1 log(1− σ(φVi(paVi ,u c Vi ; θVi))) j = 0 } ,
P (Û) := {UC ∼ Unif(0, 1) : UC ∈ U} ∪ {Gj,Vi ∼ Gumbel(0, 1) : Vi ∈ V, j ∈ {0, 1}},
(2)
where V are the nodes of G; σ : R→ (0, 1) is the sigmoid activation function; C2(G) is the set of C2-components of G; each Gj,Vi is a standard Gumbel random variable [24]; each φVi(·; θVi) is a neural net parameterized by θVi ∈ θ; paVi are the values of the parents of Vi; and u c Vi
are the values of UcVi := {UC : UC ∈ U s.t. Vi ∈ C}. The parameters θ are not yet specified and must be learned through training to enforce L1-consistency (Def. 4).
Let Uc and G denote the latent C2-component variables and Gumbel random variables, respectively. To estimate P M̂ (v) and P M̂ (y | do(x)) given Eq. 2, we may compute the probability mass of a datapoint v with intervention do(X = x) (X is empty when observational) as:
P M̂(G;θ)(v | do(x)) = E P (uc)  ∏ Vi∈V\X σ̃vi  ≈ 1 m m∑ j=1 ∏ Vi∈V\X σ̃vi , (3)
where σ̃vi := { σ(φi(paVi ,u c Vi ; θVi)) vi = 1
1− σ(φi(paVi ,u c Vi ; θVi)) vi = 0 and {ucj}mj=1 are samples from P (Uc). Here,
we assume v is consistent with x (the values of X ∈ X in v match the corresponding ones of x). Otherwise, P M̂(G;θ)(v | do(x)) = 0. For numerical stability of each φi(·), we work in log-space and use the log-sum-exp trick.
Algorithm 2: Training Model
Input : Data {vk}nk=1, variables V, X ⊆ V, x ∈ DX, Y ⊆ V, y ∈ DY , causal diagram G, number of Monte Carlo samplesm, regularization constant λ, learning rate η
1 M̂ ← NCM(V,G) // from Def. 7 2 Initialize parameters θmin and θmax 3 for k ← 1 to n do // Estimate from Eq. 3 4 p̂min ← Estimate(M̂(θmin),V,vk, ∅, ∅,m) 5 p̂max ← Estimate(M̂(θmax),V,vk, ∅, ∅,m) 6 q̂min ← 0 7 q̂max ← 0 8 for v ∈ DV do 9 if Consistent(v,y) then 10 q̂min ← q̂min+ Estimate(M̂(θmin),V,v,X,x,m) 11 q̂max ← q̂max+ Estimate(M̂(θmax),V,v,X,x,m)
// L from Eq. 5 12 Lmin ← − log p̂min − λ log(1− q̂min) 13 Lmax ← − log p̂max − λ log q̂max 14 θmin ← θmin + η∇Lmin 15 θmax ← θmax + η∇Lmax
Alg. 1 (lines 2-3) requires non-trivial evaluations of expressions like arg maxθ P M̂ (y | do(x)) while enforcing L1-consistency. Whenever only finite samples are available {vk}nk=1 ∼ P ∗(V), the parameters of an L1-consistent NCM may be estimated by minimizing data negative log-likelihood:
θ ∈ arg min θ
EP∗(v) [ − logP M̂(G;θ)(v) ] ≈ arg min
θ
1
n n∑ k=1 − log P̂ M̂(G;θ)m (vk). (4)
To simultaneously maximize P M̂ (y | do(x)), we subtract a weighted second term log P̂ M̂m (y | do(x)), resulting in the objective L({vk}nk=1) equal to
1
n n∑ k=1 − log P̂ M̂m (vk)− λ log P̂ M̂m (y | do(x)), (5)
where λ is initially set to a high value and decreases during training. To minimize, we instead subtract λ log(1− P̂ M̂m (y | do(x))) from the log-likelihood.
Alg. 2 is one possible way of optimizing the parameters θ required in lines 2,3 of Alg. 1. Eq. 5 is amenable to optimization through standard gradient descent tools, e.g., [38, 51, 50]. 9 10
One way of understanding Alg. 1 is as a search within the Ω(G) space for two NCM parameterizations, θ∗min and θ ∗ max, that minimizes/maximizes the interventional distribution, respectively. Whenever the optimization ends, we can compare the corresponding P (y | do(x)) and determine whether an effect is identifiable. With perfect optimization and unbounded resources, identifiability entails the equality between these two quantities. In practice, we rely on a hypothesis testing step such as
|f(M̂(θmax))− f(M̂(θmin))| < τ (6)
for quantity of interest f and a certain threshold τ . This threshold is somewhat similar to a significance level in statistics and can be used to control certain types of errors. In our case, the threshold τ can be determined empirically. For further discussion, see Appendix B.

heading: 5 Experiments
text: We start by evaluating NCMs (following Eq. 2) in their ability to decide whether an effect is identifiable through Alg. 2. Observational data is generated from 8 different SCMs, and their corresponding causal diagrams are shown in Fig. 4 (top part), and Appendix B provides further details of the parametrizations. Since the NCM does not have access to the true SCM, the causal diagram and generated datasets are passed to the algorithm to decide whether an effect is identifiable. The target effect is P (Y | do(X)), and the quantity we optimize is the average treatment effect (ATE) of X on Y , ATEM(X,Y ) = EM[Y | do(X = 1)] − EM[Y | do(X = 0)]. Note that if the outcome Y is binary, as in our examples, E[Y | do(X = x)] = P (Y = 1|do(X = x)). The effect is identifiable through do-calculus in the settings represented by Fig. 4 in the left part, and not identifiable in right.
The bottom row of Fig. 4 shows the max-min gaps, the l.h.s of Eq. 6 with f(M) = ATEM(X,Y ), over 3000 training epochs. The parameter λ is set to 1 at the beginning, and decreases logarithmically over each epoch until it reaches 0.001 at the end of training. The max-min gaps can be used to classify the quantity as “ID” or “non-ID” using the hypothesis testing procedure described in Appendix B. The classification accuracies per training epoch are shown in Fig. 4 (middle row). Note that in identifiable settings, the gaps slowly reduce to 0, while the gaps rapidly grow and stay high throughout training in the unidentifiable ones. The classification accuracy for ID cases then gradually increases as training
9Our approach is flexible and may take advantage of these different methods depending on the context. There are a number of alternatives for minimizing the discrepancy between P ∗ and P M̂ , including minimizing divergences, such as maximum mean discrepancy [23] or kernelized Stein discrepancy [49], performing variational inference [9], or generative adversarial optimization [20].
10The NCM can be extended to the continuous case by replacing the Gumbel-max trick on σ(φi(·)) with a model that directly computes a probability density given a data point, e.g., normalizing flow [62] or VAE [39].
progresses, while accuracy for non-ID cases remain high the entire time (perfect in the bow and IV cases).
In the identifiable settings, we also evaluate the performance of the NCM at estimating the correct causal effect, as shown in Fig. 5. As a generative model, the NCM is capable of generating samples from both P (V) and identifiable L2 distributions like P (Y | do(X)). We compare the NCM to a naïve generative model trained via likelihood maximization fitted on P (V) without using the inductive bias of the NCM. Since the naïve model is not defined to sample from P (y | do(x)), this shows the implications of arbitrarily choosing P (y | do(x)) = P (y | x). Both models improve at fitting P (V) with more samples, but the naïve model fails to learn the correct ATE except in case (c), where P (y | do(x)) = P (y | x). Further, the NCM is competitive with WERM [33], a state-of-the-art estimation method that directly targets estimating the causal effect without generating samples.

heading: 6 Conclusions
text: In this paper, we introduced neural causal models (NCMs) (Def. 3, 18), a special class of SCMs trainable through gradient-based optimization techniques. We showed that despite being as expressive as SCMs (Thm. 1), NCMs are unable to perform cross-layer inferences in general (Corol. 1). Disentangling expressivity and learnability, we formalized a new type of inductive bias based on nonparametric, structural properties of the generating SCM, accompanied with a constructive procedure that allows NCMs to represent constraints over the space of interventional distributions akin to causal diagrams (Thm. 2). We showed that NCMs with this bias retain their full expressivity (Thm. 3) but are now empowered to solve canonical tasks in causal inference, including the problems of identification and estimation (Thm. 4). We grounded these results by providing a training procedure that is both sound and complete (Alg. 1, 2, Cor. 4). Practically speaking, different neural implementations – combination of architectures, training algorithms, loss functions – can leverage the framework results introduced in this work (Appendix D.1). We implemented one of such alternatives as a proof of concept, and experimental results support the feasibility of the proposed approach. After all, we hope the causal-neural framework established in this paper can help develop more principled and robust architectures to empower the next generation of AI systems. We expect these systems to combine the best of both worlds by (1) leveraging causal inference capabilities of processing the structural invariances found in nature to construct more explainable and generalizable decision-making procedures, and (2) leveraging deep learning capabilities to scale inferences to handle challenging, high dimensional settings found in practice.

heading: Acknowledgements
text: We thank Judea Pearl, Richard Zemel, Yotam Alexander, Juan Correa, Sanghack Lee, and Junzhe Zhang for their valuable feedback. Kevin Xia and Elias Bareinboim were supported in part by funding from the NSF, Amazon, JP Morgan, and The Alfred P. Sloan Foundation. Yoshua Bengio was supported in part by funding from CIFAR, NSERC, Samsung, and Microsoft.

Review summaries:
The paper considers a restricted class of SCMs (structural causal models) where each variable in a graph is generated by applying a neural-network-parameterized function of the variable's parents, called Neural Causal Models (NCMs). The paper shows a wealth of results mapping the relationship between the full class of SCMs and the smaller class of NCMs in terms of when and how identification holds and how estimation of the interventional distribution can be achieved. The method proposed to identify and estimate effects takes a graph as an input, computes the maximum and minimum of the effect of a particular intervention within the class of NCMs that are consistent with the observed data distribution (at the lowest level in the causal hierarchy, observational); if the maximum and minimum match, the effect is identified. The method is evaluated on a few experiments.

The authors investigate the connection between structural causal models and neural networks from an learning perspective. Their theoretical results emphasize, notably, that the impossibility to learn neural networks that are interventionally equivalent to the true SCM based on observations only. The paper provides an interesting perspective on the connection between causal inference and learning with neural network, although clarity could be improved, notably regarding the limitations and the contribution with respect to previous work.

This paper studies the expressivity, identification, estimation of causality with neural models. For expressiveness, it shows that Neural Causal Model (NCM) is expressive enough to represent any structural causal model (SCM), but in general, given observation data alone, without further constraints, the NCM is not able to identify higher layers of Pearl Causal Hierarchy (PCH). The authors then develop a necessary and sufficient condition to determine whether a causal effect can be learned from data (i.e., causal identifiability); if identifiability holds, it also estimates the effect from data (causal estimation). Empirical experiments on a simple synthetic dataset support the theoretical results.

In this paper a subclass of SCMs whose structural equations are based on feed-forward NNs are considered. The authors show the expressiveness of this class and address the problem of training those model and evaluated identifiability of a causal task. Experiments on models (with few variables) are promising.