The Causal-Neural Connection: Expressiveness, Learnability, and Inference

1 Introduction . One of the most celebrated and relied upon results in the science of intelligence is the universality of neural models . More formally , universality says that neural models can approximate any function ( e.g. , boolean , classification boundaries , continuous valued ) with arbitrary precision given enough capacity in terms of the depth and breadth of the network [ 14 , 26 , 47 , 53 ] . This result , combined with the observation that most tasks can be abstracted away and modeled as input/output – i.e. , as functions – leads to the strongly held belief that under the right conditions , neural networks can solve the most challenging and interesting tasks in AI . This belief is not without merits , and is corroborated by ample evidence of practical successes , including in compelling tasks in computer vision [ 43 ] , speech recognition [ 22 ] , and game playing [ 54 ] . Given that the universality of neural nets is such a compelling proposition , we investigate this belief in the context of causal reasoning . To start understanding the causal-neural connection – i.e. , the non-trivial and somewhat intricate relationship between these modes of reasoning – two standard objects in causal analysis will be instrumental . First , we evoke a class of generative models known as the Structural Causal Model ( SCM , for short ) [ 58 , Ch . 7 ] . In words , an SCMM∗ is a representation of a system that includes a collection of mechanisms and a probability distribution over the exogenous conditions ( to be formally defined later on ) . Second , any fully specified SCMM∗ induces a collection of distributions known as the Pearl Causal Hierarchy ( PCH ) [ 5 , Def . 9 ] . The importance of the PCH is that it formally delimits 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . distinct cognitive capabilities ( also known as layers ; not to be confused with neural nets layers ) that can be associated with the human activities of “ seeing ” ( layer 1 ) , “ doing ” ( 2 ) , and “ imagining ” ( 3 ) [ 59 , Ch . 1 ] . 1 Each of these layers can be expressed as a distinct formal language and represents queries that can help to classify different types of inferences [ 5 , Def . 8 ] . Together , these layers form a strict containment hierarchy [ 5 , Thm . 1 ] . We illustrate these notions in Fig . 1 ( a ) ( left side ) , where SCMM∗ induces layers L∗1 , L∗2 , L∗3 of the PCH . Even though each possible statement within these capabilities has well-defined semantics given the true SCM M∗ [ 58 , Ch . 7 ] , a challenging inferential task arises when one wishes to recover part of the PCH whenM∗ is only partially observed . This situation is typical in the real world aside from some special settings in physics and chemistry where the laws of nature are understood with high precision . For concreteness , consider the setting where one needs to make a statement about the effect of a new intervention ( i.e. , about layer 2 ) , but only has observational data from layer 1 , which is passively collected.2 Going back to the causalneural connection , one could try to learn a neural model N using the observational dataset ( layer 1 ) generated by the true SCMM∗ , as illustrated in Fig . 1 ( b ) . Naturally , a basic consistency requirement is that N should be capable of generating the same distributions asM∗ ; in this case , their layer 1 predictions should match ( i.e. , L1 = L∗1 ) . Given the universality of neural models , it is not hard to believe that these constraints can be satisfied in the large sample limit . The question arises of whether the learned model N can act as a proxy , having the capability of predicting the effect of interventions that matches the L2 distribution generated by the true ( unobserved ) SCMM∗ . 3 The answer to this question can not be ascertained in general , as will become evident later on ( Corol . 1 ) . The intuitive reason behind this result is that there are multiple neural models that are equally consistent w.r.t . the L1 distribution ofM∗ but generate different L2-distributions . 4 Even though N may be expressive enough to fully representM∗ ( as discussed later on ) , generating one particular parametrization of N consistent with L1 is insufficient to provide any guarantee regarding higher-layer inferences , i.e. , about predicting the effects of interventions ( L2 ) or counterfactuals ( L3 ) . The discussion above entails two tasks that have been acknowledged in the literature , namely , causal effect identification and estimation . The first – causal identification – has been extensively studied , and general solutions have been developed , such as Pearl ’ s celebrated do-calculus [ 57 ] . Given the impossibility described above , the ingredient shared across current non-neural solutions is to represent assumptions about the unknownM∗ in the form of causal diagrams [ 58 , 65 , 7 ] or their equivalence classes [ 28 , 60 , 29 , 71 ] . The task is then to decide whether there is a unique solution for the causal query based on such assumptions . There are no neural methods today focused on solving this task . The second task – causal estimation – is triggered when effects are determined to be identifiable by the first task . Whenever identifiability is obtained through the backdoor criterion/conditional ignorability [ 58 , Sec . 3.3.1 ] , deep learning techniques can be leveraged to estimate such effects with impressive practical performance [ 63 , 52 , 48 , 31 , 69 , 70 , 35 , 64 , 15 , 25 , 37 , 30 ] . For effects that are identifiable through causal functionals that are not necessarily of the backdoor-form ( e.g. , frontdoor , 1This structure is named after Judea Pearl and is a central topic in his Book of Why ( BoW ) , where it is also called the “ Ladder of Causation ” [ 59 ] . For a more technical discussion on the PCH , we refer readers to [ 5 ] . 2The full inferential challenge is , in practice , more general since an agent may be able to perform interventions and obtain samples from a subset of the PCH ’ s layers , while its goal is to make inferences about some other parts of the layers [ 7 , 46 , 5 ] . This situation is not uncommon in RL settings [ 66 , 17 , 44 , 45 ] . Still , for the sake of space and concreteness , we will focus on two canonical and more basic tasks found in the literature . 3We defer a more formal discussion on how neural models could be used to assess the effect of interventions to Sec . 2 . Still , this is neither attainable in all universal neural architectures nor trivially implementable . 4Pearl shared a similar observation in the BoW [ 59 , p. 32 ] : “ Without the causal model , we could not go from rung ( layer ) one to rung ( layer ) two . This is why deep-learning systems ( as long as they use only rung-one data and do not have a causal model ) will never be able to answer questions about interventions ( ... ) ” . napkin ) , other optimization/statistical techniques can be employed that enjoy properties such as double robustness and debiasedness [ 32 , 33 , 34 ] . Each of these approaches optimizes a particular estimand corresponding to one specific target interventional distribution . Despite all the great progress achieved so far , it is still largely unknown how to perform the tasks of causal identification and estimation in arbitrary settings using neural networks as a generative model , acting as a proxy for the true SCMM∗ . It is our goal here to develop a general causal-neural framework that has the potential to scale to real-world , high-dimensional domains while preserving the validity of its inferences , as in traditional symbolic approaches . In the same way that the causal diagram encodes the assumptions necessary for the do-calculus to decide whether a certain query is identifiable , our method encodes the same invariances as an inductive bias while being amenable to gradient-based optimization , allowing us to perform both tasks in an integrated fashion ( in a way , addressing Pearl ’ s concerns alluded to in Footnote 4 ) . Specifically , our contributions are as follows : 1 . [ Sec . 2 ] We introduce a special yet simple type of SCM that is amenable to gradient descent called a neural causal model ( NCM ) . We prove basic properties of this class of models , including its universal expressiveness and ability to encode an inductive bias representing certain structural invariances ( Thm . 1-3 ) . Notably , we show that despite the NCM ’ s expressivity , it still abides by the Causal Hierarchy Theorem ( Corol . 1 ) . 2 . [ Sec . 3 ] We formalize the problem of neural identification ( Def . 8 ) and prove a duality between identification in causal diagrams and in neural causal models ( Thm . 4 ) . We introduce an operational way to perform inferences in NCMs ( Corol . 2-3 ) and a sound and complete algorithm to jointly train and decide effect identifiability for an NCM ( Alg . 1 , Corol . 4 ) . 3 . [ Sec . 4 ] Building on these results , we develop a gradient descent algorithm to jointly identify and estimate causal effects ( Alg . 2 ) . There are multiple ways of grounding these theoretical results . In Sec . 5 , we perform experiments with one possible implementation which support the feasibility of the proposed approach . All appendices including proofs , experimental details , and examples can be found in the full technical report [ 68 ] . 1.1 Preliminaries . In this section , we provide the necessary background to understand this work , following the presentation in [ 58 ] . An uppercase letter X indicates a random variable , and a lowercase letter x indicates its corresponding value ; bold uppercase X denotes a set of random variables , and lowercase letter x its corresponding values . We use DX to denote the domain of X and DX = DX1 × · · · × DXk for X = { X1 , . . . , Xk } . We denote P ( X ) as a probability distribution over a set of random variables X and P ( X = x ) as the probability of X being equal to the value of x under the distribution P ( X ) . For simplicity , we will mostly abbreviate P ( X = x ) as simply P ( x ) . The basic semantic framework of our analysis rests on structural causal models ( SCMs ) [ 58 , Ch . 7 ] , which are defined below . Definition 1 ( Structural Causal Model ( SCM ) ) . An SCMM is a 4-tuple 〈U , V , F , P ( U ) 〉 , where U is a set of exogenous variables ( or “ latents ” ) that are determined by factors outside the model ; V is a set { V1 , V2 , . . . , Vn } of ( endogenous ) variables of interest that are determined by other variables in the model – that is , in U ∪V ; F is a set of functions { fV1 , fV2 , . . . , fVn } such that each fi is a mapping from ( the respective domains of ) UVi ∪PaVi to Vi , where UVi ⊆ U , PaVi ⊆ V \ Vi , and the entire set F forms a mapping from U to V. That is , for i = 1 , . . . , n , each fi ∈ F is such that vi ← fVi ( paVi , uVi ) ; and P ( u ) is a probability function defined over the domain of U . Each SCMM induces a causal diagram G where every Vi ∈ V is a vertex , there is a directed arrow ( Vj → Vi ) for every Vi ∈ V and Vj ∈ Pa ( Vi ) , and there is a dashed-bidirected arrow ( Vj L9999K Vi ) for every pair Vi , Vj ∈ V such that UVi and UVj are not independent . For further details on this construction , see [ 5 , Def . 13/16 , Thm . 4 ] . The exogenous UVi ’ s are not assumed independent ( i.e . Markovianity does not hold ) . We will consider here recursive SCMs , which implies acyclic diagrams , and that the endogenous variables ( V ) are discrete and have finite domains . We show next how an SCMM gives values to the PCH ’ s layers ; for details on the semantics , see [ 5 , Sec . 1.2 ] . Superscripts are omitted when unambiguous . Definition 2 ( Layers 1 , 2 Valuations ) . An SCMM induces layer L2 ( M ) , a set of distributions over V , one for each intervention x . For each Y ⊆ V , PM ( yx ) = ∑ { u|Yx ( u ) =y } P ( u ) , ( 1 ) where Yx ( u ) is the solution for Y after evaluatingFx : = { fVi : Vi ∈ V\X } ∪ { fX ← x : X ∈ X } . The specific distribution P ( V ) , where X is empty , is defined as layer L1 ( M ) . In words , an external intervention forcing a set of variables X to take values x is modeled by replacing the original mechanism fX for each X ∈ X with its corresponding value in x . This operation is represented formally by the do-operator , do ( X = x ) , and graphically as the mutilation procedure . For the definition of the third layer , L3 ( M ) , see Def . 9 in Appendix A or [ 5 , Def . 7 ] .

Summary: In this paper a subclass of SCMs whose structural equations are based on feed-forward NNs are considered. The authors show the expressiveness of this class and address the problem of training those model and evaluated identifiability of a causal task. Experiments on models (with few variables) are promising.