Dynamic Causal Bayesian Optimization

1 Introduction Causal Dimension Solving decision making problems in a variety of domains requires understanding of cause-effect relationships in a system . This can be obtained by experimentation . However , deciding how to intervene at every point in time is particularly complex in dynamical systems , due to the evolving nature of causal effects . For instance , companies need to decide how to allocate scarce resources across different quarters . In system biology , scientists need to identify genes to knockout at specific points in time . This paper describes a probabilistic framework that finds such optimal interventions over time . Focusing on a specific example , consider a setting in which Yt denotes the unemployment-rate of an economy at time t , Zt is the economic growth and Xt the inflation rate . Fig . 1a depicts the causal graph [ 26 ] representing an agent ’ s understanding of the causal links between these variables . The agent aims to determine , at each time step t ∈ { 0 , 1 , 2 } , the optimal action to perform in order to minimize the current unemployment rate ∗Denotes equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Yt while accounting for the intervention cost . The investigator could frame this setting as a sequence of global optimization problems and find the solutions by resorting to Causal Bayesian Optimization [ CBO , 2 ] . CBO extends Bayesian Optimization [ BO , 30 ] to cases in which the variable to optimize is part of a causal model where a sequence of interventions can be performed . However , CBO does not account for the system ’ s temporal evolution thus breaking the time dependency structure existing among variables ( Fig . 1b ) . This will lead to sub-optimal solutions , especially in non-stationary scenarios . The same would happen when using Adaptive Bayesian Optimization [ ABO , 25 ] ( Fig . 1c ) or BO ( Fig . 1d ) . ABO captures the time dependency of the objective function but neither considers the causal structure among inputs nor their temporal evolution . BO disregards both the temporal and the causal structure . Our setting differs from both reinforcement learning ( RL ) and the multi-armed bandits setting ( MAB ) . Differently from MAB we consider interventions on continuous variables where the dynamic target variable has a non-stationary interventional distribution . In addition , compared to RL , we do not model the state dynamics explicitly and allow the agent to perform a number of explorative interventions which do not change the underlying state of the system , before selecting the optimal action . We discuss these points further in §1.1 . Dynamic Causal Bayesian Optimization2 , henceforth DCBO , accounts for both the causal relationships among input variables and the causality between inputs and outputs which might evolve over time . DCBO integrates CBO with dynamic Bayesian networks ( DBN ) , offering a novel approach for decision making under uncertainty within dynamical systems . DBN [ 19 ] are commonly used in time-series modelling and carry dependence assumptions that do not imply causation . Instead , in probabilistic causal models [ 27 ] , which form the basis for the CBO framework , graphs are buildt around causal information and allow us to reason about the effects of different interventions . By combining CBO with DBNs , the proposed methodology finds an optimal sequence of interventions which accounts for the causal temporal dynamics of the system . In addition , DCBO takes into account past optimal interventions and transfers this information across time , thus identifying the optimal intervention faster than competing approaches and at a lower cost . We make the following contributions : • We formulate a new class of optimization problems called Dynamic Causal Global Optimization ( DCGO ) where the objective functions account for the temporal causal dynamics among variables . • We give theoretical results demonstrating how interventional information can be transferred across time-steps depending on the topology of the causal graph . • Exploiting our theoretical results , we solve the optimization problem with DCBO . At every time step , DCBO constructs surrogate models for different intervention sets by integrating various sources of data while accounting for past interventions . • We analyze DCBO performance in a variety of settings comparing against CBO , ABO and BO . 1.1 Related Work . Dynamic Optimization Optimization in dynamic environments has been studied in the context of evolutionary algorithms [ 14 , 16 ] . More recently , other optimization techniques [ 28 , 32 , 10 ] have been adapted to dynamic settings , see e.g . [ 9 ] for a review . Focusing on BO , the literature on dynamic settings [ 3 , 7 , 25 ] is limited . The dynamic BO framework closest to this work is given by Nyikosa et al . [ 25 ] and focuses on functions defined on continuous spaces that follow a more complex behaviour than a simple Markov model . ABO treats the inputs as fixed and not as random variables , thereby disregarding their temporal evolution and , more importantly , breaking their causal dependencies . Causal Optimization Causal BO [ CBO , 2 ] focuses instead on the causal aspect of optimization and solves the problem of finding an optimal intervention in a DAG by modelling the intervention functions with single GPs or a multi-task GP model [ 1 ] . CBO disregards the existence of a temporal evolution in both the inputs and the output variable , treating them as i.i.d . overtime . While disregarding time significantly simplifies the problem , it prevents the identification of an optimal intervention at every t. Bandits and RL In the broader decision-making literature , causal relationships have been considered in the context of bandits [ 4 , 20–22 ] and reinforcement learning [ 23 , 8 , 13 , 36 , 24 ] . In these cases , actions or arms , correspond to interventions on a causal graph where there exists complex relationships between the agent ’ s decisions and the received rewards . While dynamic settings have been considered in acausal bandit algorithms [ 5 , 33 , 35 ] , causal MAB have focused on static settings . Dynamic settings are instead considered by RL algorithms and formalized through Markov decision processes ( MDP ) . 2A Python implementation is available at : https : //github.com/neildhir/DCBO . In the current formulation , DCBO does not consider an MDP as we do not have a notion of state and therefore do not require an explicit model of its dynamics . The system is fully specified by the causal model . As in BO , we focus on identifying a set of time-indexed optimal actions rather than an optimal policy . We allow the agent to perform explorative interventions that do not lead to state transitions . More importantly , differently from both MAB and RL , we allow for the integration of both observational and interventional data . An expanded discussion on the reason why DCBO should be used and the links between DCBO , CBO , ABO and RL is included in the supplement ( §8 ) . 2 Background and Problem Statement . Let random variables and values be denoted by upper-case and lower-case letters respectively . Vectors are represented shown in bold . do ( X = x ) represents an intervention on X whose value is set to x. p ( Y | X = x ) represents an observational distribution and p ( Y | do ( X = x ) ) represents an interventional distribution . This is the distribution of Y obtained by intervening on X and fixing its value to x in the data generating mechanism ( see Fig . 2 ) , irrespective of the values of its parents . Evaluating p ( Y | do ( X = x ) ) requires “ real ” interventions while p ( Y | X = x ) only requires “ observing ” the system . DO and DI denote observational and interventional datasets respectively . Consider a structural causal model ( SCM ) defined in Definition 1 . Definition 1 . ( Structural Causal Model ) [ 27 , p. 203 ] . A structural causal model M is a triple 〈U , V , F ) 〉 where U is a set of background variables ( also called exogenous ) , that are determined by factors outside of the model . V is a set { V1 , V2 , . . . , V|V| } of observable variables ( also called endogenous ) , that are determined by variables in the model ( i.e. , determined by variables in U ∪V ) . F is a set of functions { f1 , f2 , . . . , fn } such that each fi is a mapping from the respective domains of Ui ∪ Pa ( Vi ) to Vi , where Ui ⊆ U and Pa ( Vi ) ⊆ V \ Vi and the entire set F forms a mapping from U to V. In other words , each { fi ∈ vi ← fi ( Pa ( vi ) , ui ) | i = 1 , . . . , n } , assigns a value to Vi that depends on the values of the select set of variables ( Ui ∪ Pa ( Vi ) ) . M is associated to a directed acyclic graph ( DAG ) G , in which each node corresponds to a variable and the directed edges point from members of Pa ( Vi ) and Ui to Vi . We assume G to be known and leave the integration with causal discovery [ 15 ] methods for future work . Within V , we distinguish between three different types of variables : non-manipulative variables C , which can not be modified , treatment variables X that can be set to specific values and output variable Y that represents the agent ’ s outcome of interest . Exploiting the rules of do-calculus [ 27 ] one can compute p ( Y | do ( X = x ) ) using observational data . This often involves evaluating intractable integrals which can be approximated by using observational data to get a Monte Carlo estimate p̂ ( Y | do ( X = x ) ) ≈ p ( Y | do ( X = x ) ) . These approximations will be consistent when the number of samples drawn from p ( V ) is large . Causality in time One can encode the existence of causal mechanisms across time steps by explicitly representing these relationships with edges in an extended graph denoted by G0 : T . For instance , the DAG in Fig . 1 ( a ) can be seen as one of the DAGs in Fig . 1 ( b ) propagated in time . The DAG in Fig . 1 ( a ) captures both the causal structure existing across time steps and the causal mechanism within every “ time-slice ” t [ 19 ] . In order to reason about interventions that are implemented in a sequential manner , that is at time t we decide which intervention to perform in the system and so define : Definition 2 . Mt is the SCM at time step t defined asMt = 〈U0 : t , V0 : t , F0 : t〉 where 0 : t denotes the union of the corresponding variables or functions up to time t ( see Fig . 2 ) . V0 : t includes X0 : t = Xt , Y0 : t = Yt and C0 : t = Ct ∪C0 : t−1 . The functions in F0 : t corresponding to intervened variables are replaced by constant values while the exogenous variables related to them are excluded from U0 : t. Definition 3 . Gt is the causal graph associated to Mt . In Gt , the incoming edges in variables intervened at 0 : t− 1 are mutilated while intervened variables are represented by deterministic nodes ( squares ) – see Fig . 2 . Dynamic Causal Global Optimization ( DCGO ) The goal of this work is to find a sequence of interventions , optimizing a target variable , at each time step , in a causal DAG . Given Gt and Mt , at every time step t , we wish to optimize Yt by intervening on a subset of the manipulative variables Xt . The optimal intervention variables X ? s , t and intervention levels x ? s , t are given by : X ? s , t , x ? s , t = argmin Xs , t∈P ( Xt ) , xs∈D ( Xs , t ) E [ Yt | do ( Xs , t = xs , t ) ,1t > 0 · I0 : t−1 ] ( 1 ) where I0 : t−1 = ⋃t−1 i=0 do ( X ? s , i = x ? s , i ) denotes previous interventions , 1t > 0 is the indicator function and P ( Xt ) is the power set of Xt . D ( Xs , t ) represents the interventional domain of Xs , t . In the sequel we denote the previously intervened variables by IV0 : t−1 = ⋃t−1 i=0 X ? s , i and implemented intervention levels by IL0 : t−1 = ⋃t−1 i=0 x ? s , i . The cost of each intervention is given by cost ( Xs , t , xs , t ) . In order to solve the problem in Eq . ( 1 ) we make the following assumptions : Assumptions 1 . Denote by G ( t ) the causal graph including variables at time t in G0 : T and let Y PTt = Pa ( Yt ) ∩ Y0 : t−1 be the set of variables in G0 : T that are both parents of Yt and targets at previous time step . Let the set Y PNTt = Pa ( Yt ) \Y PTt be the complement and denote by fYt ( · ) the functional mapping for Yt in Mt . We make the following assumptions : 1 . Invariance of causal structure : G ( t ) = G ( 0 ) , ∀t > 0 . 2 . Additivity of fYt ( · ) that is Yt = fYt ( Pa ( Yt ) ) + with fYt ( Pa ( Yt ) ) = fYY ( Y PTt ) + fNYY ( Y PNTt ) where fYY and f NY Y are two generic unknown functions and ∼ N ( 0 , σ2 ) . 3 . Absence of unobserved confounders in G0 : T . Assumption ( 3 ) implies the absence of unobserved confounders at every time step . For instance , this is the case in Fig . 1a . Still in this DAG , Assumption ( 2 ) implies fYt ( Pa ( Yt ) ) = f Y Y ( Yt−1 ) + fNYY ( Zt ) + Yt , ∀t > 0 . Finally , Assumption ( 1 ) implies the existence of the same variables at every time step and a constant orientation of the edges among them for t > 0 . Notice that Assumptions 1 imply invariance of the causal structure within each time-slice , i.e . the structure , edges and vertices , concerning the nodes with the same time index . This means that , across time steps , both the graph and the functional relationships can change . Therefore , not only can the causal effects change significantly across time steps , but also the input dimensionality of the causal functions we model , might change . For instance , in the DAG of Fig . 3 ( c ) , the target function for Y2 has dimensionality 3 and a function fYt ( · ) that is completely different from the one assumed for Y1 that has only two parents . We can thus model a wide variety of settings and causal effects despite this assumption . Furthermore , even though we assume an additive structure for the functional relationship on Y , the use of GPs allow us to have flexible models with highly non-linear causal effects across different graph structures . In the causality literature , GP models are well established and have shown good performances compared to parametric linear and non-linear models ( see e.g . [ 31 , 34 , 37 ] ) . The sum of GPs gives a flexible and computationally tractable model that can be used to capture highly non-linear causal effects while helping with interpretability [ 12 , 11 ] .

Summary: This paper works with the Dynamic Causal Global Optimization problem. For a temporal structural causal model, which is a sequence of structural causal models with causality in time, meaning there are dependencies on the values of variables from previous step. The goal is to optimize the outcome variable by manipulating some observable variables (those are chosen by the model).  An algorithm called dynamic causal bayesian optimization is proposed to solve such a problem. The objective function $f_{s,t}$ is decomposed into the sum of a function depending on the optimized observed variables and another function depending on others (variables in the current time). Thus the optimization can be done step by step in time.  A dynamic causal GP model is introduced for practically implement the DCBO algorithm.  For experiments, both Synthetic and Real data are used, in several different structural causal models, to compare the behavior of DCBO, to that of CBO, ABO and BO.