Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models

1 Introduction . Identifying causal relationships between variables in observational data is one of the fundamental and well-studied problem in machine learning . There have been several great strides in causality [ Granger , 1969 , Pearl , 2009 , Bareinboim and Pearl , 2016 ] over the years specifically characterized by efforts that focused on reasoning about interventions [ Hagmayer et al. , 2007 , Dasgupta et al. , 2019 ] and counterfactuals [ Morgan and Winship , 2015 , Oberst and Sontag , 2019 ] . The notion of causality has long been explored in the realm of probabilistic models [ Oaksford and Chater , 2017 , Beckers and Halpern , 2019 ] with a special focus on graphical models , called causal Bayesian networks ( CBNs ) [ Heckerman et al. , 1995 , Neapolitan , 2004 , Pearl , 1995 , Acharya et al. , 2018 ] . CBNs have widely been applied to infer causal relationships in high-impact diverse applications such as disease progression [ Koch et al. , 2017 ] , ecological risk assessment [ Carriger and Barron , 2020 ] and more recently Covid-19 [ Fenton et al. , 2020 , Feroze , 2020 ] to name a few . Although successful , classical CBN models are difficult to scale and also suffer from the problem of intractable inference . Recently , tractable probabilistic models such as probabilistic sentential decision diagrams [ Kisa et al. , 2014 ] and sum-product networks [ Poon and Domingos , 2011 ] have emerged , which guarantee that conditional marginals can be computed in time linear in the size of the model . While weaving in the notion of interpretability , the computational view on probabilistic models allows one to exploit ideas from deep learning and can thus be very useful in modelling complex problems . Recently , there has been an effort to take advantage of this tractability to reason for causality . Zhao et al . [ 2015 ] showed how to compile back and forth between sum-product networks ( SPNs ) and Bayesian networks ( BNs ) . Although this opened up a whole range of possibilities for tractable 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . causal models , Papantonis and Belle [ 2020 ] argued that such conversion leads to degenerated BNs thereby rendering it ineffective for causal reasoning . For the considered compilation of SPNs to BNs , this is indeed the case since a bipartite graph between the hidden and observed variables loses the relationships between the actual variables . Thus , either a new compilation method for transforming between tractable and causal model or alternatively a method to condition the probabilistic models directly on the do-operator to obtain interventional distributions , P ( y| do ( x ) ) , is being required . Here , we consider the latter strategy and extend the idea of conditionally parameterizing SPNs [ Shao et al. , 2019 ] by conditioning on the do-operator while predicting the complete set of observed variables therefore capturing the effect of intervention ( s ) . The resulting interventional sum-product networks ( iSPNs ) take advantage of both the expressivity , due to the neural network , and the tractability , due to the SPN in order to capture the interventional distributions faithfully . This shows that the dream of tractable causal models is not insurmountable , since iSPNs are causal . Pearl [ 2019 ] defined a three-level causal hierarchy that separates association ( purely statistical level 1 ) from intervention ( level 2 ) and counterfactuals ( level 3 ) , and argued that the latter two levels involve causal inference . iSPNs are a modelling scheme for arbitrary interventional distributions . They thus belong to level 2 and , in turn , are causal . So , while SPNs are not universal function approximators and the use of gate functions turns them into universal approximators , we go one step ahead and make the first effort towards introducing causality to SPNs without the need for compilation to Bayesian networks , as the functional approximator subsumes the do-operator . Fig . 1 shows an example of the effectiveness of iSPNs to capture interventional distributions on the ASIA data set . Our extensive experiments against strong baselines demonstrate that our method is able to capture ideal interventional distributions . To summarize , we make the following contributions : 1 . We introduce iSPNs , the first method that applies the idea of tractable probabilistic models to causality without the need for compilation and that accordingly generate interventional distributions w.r.t . the provided causal structure . 2 . We formulate the inductive bias necessary for turning conditional SPN into iSPN while taking advantage of the neural network modelling capacities within the gating nodes for for modelling interventional distributions while capturing all influences within the given intervention ( i.e. , consequences propagated through the structural hierarchy ) . 3 . We show that , by construction , iSPNs can identify any interventional distribution permitted by the underlying structural causal model due to the inducted bias on the interface modalities in junction with the universal function approximation of the underlying gated SPN . We proceed as follows . We start by reviewing the basic concepts required and related work , namely the tractable model class of sum-product networks and key concepts from causality . Then we motivate using a newly curated causal model themed around personal health . Subsequently , we introduce iSPNs formally and prove that by construction they are capable of approximating any do-query ( given corresponding data ) . Before concluding , we present our experimental evaluation in which we challenge iSPN in its density estimation and causal inference capabilities against various baselines . We make our code publically available at : https : //github.com/zecevic-matej/iSPN . 2 Background and Related Work . Let us briefly review the background on tractable probabilistic models and causal models used in subsequent sections for developing our new model class based on CSPNs that allow for identifying causal quantities i.e. , interventional distributions . Notation . We denote indices by lower-case letters , functions by the general form g ( · ) , scalars or random variables interchangeably by upper-case letters , vectors , matrices and tensors with different boldface font v , V , V respectively , and probabilities of a set of random variables X as p ( X ) . Sum-Product Networks ( SPNs ) . Introduced by Poon and Domingos [ 2011 ] , generalizing the notion of network polynomials based on indicator variables λX=x ( x ) ∈ [ 0 , 1 ] for ( finite-state ) RVs X from [ Darwiche , 2003 ] , Sum-Product Networks ( SPNs ) represent a special type of probabilistic model that allows for a variety of exact and efficient inference routines . Generally , SPNs are considered as directed acyclic graphs ( DAG ) consisting of product , sum and leaf ( or distribution ) nodes whose structure and parameterization can be efficiently learned from data to allow for efficient modelling of joint probability distributions p ( X ) . Formally a SPN S = ( G , w ) consists of non-negative parameters w and a DAG G = ( V , E ) with indicator variable λ leaf nodes and exclusively internal sum and product nodes given by , S ( λ ) = ∑ C∈ch ( S ) wS , CC ( λ ) P ( λ ) = ∏ C∈ch ( S ) C ( λ ) , ( 1 ) where the SPN output S is computed at the root node ( S ( λ ) = S ( x ) ) and the probability density for x is p ( x ) = S ( x ) ∑ x′∈X S ( x′ ) . They are members of the family of probabilistic circuits [ Van den Broeck et al. , 2019 ] . A special class , to be precise , that satisfies properties known as completeness and decomposability . Let N denote a node in SPN S , then sc ( N ) = { { X } if N is IV ( λX=x ) ⋃ C∈ch ( N ) sc ( C ) else ( 2 ) is called the scope of N and ∀S ∈ S : ( ∀C1 , C2 ∈ ch ( S ) : sc ( C1 ) = sc ( C2 ) ) ( 3 ) ∀P ∈ S : ( ∀C1 , C2 ∈ ch ( S ) : C1 ̸= C2 =⇒ sc ( C1 ) ∩ sc ( C2 ) = ∅ ) ( 4 ) are the completeness and decomposability properties respectively . Since their introduction , SPNs have been heavily studied such as by [ Trapp et al. , 2019 ] that present a way to learn SPNs in a Bayesian realm whereas [ Kalra et al. , 2018 ] learn SPNs in an online setting . Several different types of SPNs have also been studied such as Random SPN [ Peharz et al. , 2020 ] , Credal SPNs [ Levray and Belle , 2020 ] and Sum-Product-Quotient Networks [ Sharir and Shashua , 2018 ] ) to name a few . For more details readers are referred to the survey of París , Sánchez-Cauce , and Díez [ 2020 ] . On another note , Gated or Conditional SPNs ( CSPNs ) are deep tractable models for estimating multivariate , conditional probability distributions p ( Y|X ) over mixed variables Y [ Shao et al. , 2019 ] . They introduce functional gate nodes gi ( X ) that act as a functional parameterization of the SPN ’ s information flow and leaf distributions given the provided evidence X. Causal Models . A Structural Causal Model ( SCM ) as defined by Peters et al . [ 2017 ] is specified as C : = ( S , PN ) where PN is a product distribution over noise variables and S is defined to be a set of d structural equations Xi : = fi ( pa ( Xi ) , Ni ) , where i = 1 , . . . , d ( 5 ) with pa ( Xi ) representing the parents of Xi in graph G ( C ) . An intervention on a SCM C as defined in ( 5 ) occurs when ( multiple ) structural equations are being replaced through new non-parametric functions f̂ ( p̂a ( Xi ) , N̂i ) thus effectively creating an alternate SCM Ĉ . Interventions are referred to as imperfect if p̂a ( Xi ) = pa ( Xi ) and as atomic if f̂ = a for a ∈ R. An important property of interventions often referred to as `` modularity '' or `` autonomy '' 1 states that interventions are fundamentally of local nature , formally pC ( Xi | pa ( Xi ) ) = pĈ ( Xi | pa ( Xi ) ) , ( 6 ) where the intervention of Ĉ occured on variable Xk opposed to Xi . This suggests that mechanisms remain invariant to changes in other mechanisms which implies that only information about the effective changes induced by the intervention need to be compensated for . An important consequence of autonomy is the truncated factorization p ( V ) = ∏ i/∈S p ( Xi | pa ( Xi ) ) ( 7 ) derived by Pearl [ 2009 ] , which suggests that an intervention S introduces an independence of an intervened node Xi to its causal parents . Another important assumption in causality is that causal mechanisms do not change through intervention suggesting a notion of invariance to the cause-effect relations of variables which further implies an invariance to the origin of the mechanism i.e. , whether it occurs naturally or through means of intervention [ Pearl et al. , 2016 ] . A SCM C is capable of emitting various mathematical objects such as graph structure , statistical and causal quantities placing it at the heart of causal inference , rendering it applicable to machine learning applications in marketing [ Hair Jr and Sarstedt , 2021 ] ) , healthcare [ Bica et al. , 2020 ] ) and education [ Hoiles and Schaar , 2016 ] . A SCM induces a causal graphG , an observational/associational distribution pC , can be intervened upon using the do-operator and thus generate interventional distributions pC ; do ( ... ) and given some observations v can also be queried for interventions within a system with fixed noise terms amounting to counterfactual distributions pC|V=v ; do ( ... ) . To query for samples of a given SCM , the structural equations are being simulated sequentially following the underlying causal structure starting from independent , exogenous variables and then moving along the causal hierarchy of endogenous variables ( i.e. , following the causal descendants ) . The work closest to our work is by Brouillard et al . [ 2020 ] although it solves the different problem of causal discovery . Causality for machine learning has recently gained a lot of traction [ Schölkopf , 2019 ] with the study of both interventions [ Shanmugam et al. , 2015 ] and counterfactuals [ Kusner et al. , 2017 ] gaining speed . For more details the readers are referred to [ Zhang et al. , 2018 ] .

Summary: The paper proposes to learn interventional distributions using a family of generative models known as sum-product networks (SPN). SPNs are models based on gate functions that allow for tractable inference. Specifically, the authors propose interventional SPNs (iSPN) which are conditional SPNs (they model conditional distributions) that, licensed by a causal graph and samples from the interventional distribution, are shown to be expressive and causally adequate.