Dynamic Causal Bayesian Optimization

1 Introduction Causal Dimension Solving decision making problems in a variety of domains requires understanding of cause-effect relationships in a system . This can be obtained by experimentation . However , deciding how to intervene at every point in time is particularly complex in dynamical systems , due to the evolving nature of causal effects . For instance , companies need to decide how to allocate scarce resources across different quarters . In system biology , scientists need to identify genes to knockout at specific points in time . This paper describes a probabilistic framework that finds such optimal interventions over time . Focusing on a specific example , consider a setting in which Yt denotes the unemployment-rate of an economy at time t , Zt is the economic growth and Xt the inflation rate . Fig . 1a depicts the causal graph [ 26 ] representing an agent ’ s understanding of the causal links between these variables . The agent aims to determine , at each time step t ∈ { 0 , 1 , 2 } , the optimal action to perform in order to minimize the current unemployment rate ∗Denotes equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Yt while accounting for the intervention cost . The investigator could frame this setting as a sequence of global optimization problems and find the solutions by resorting to Causal Bayesian Optimization [ CBO , 2 ] . CBO extends Bayesian Optimization [ BO , 30 ] to cases in which the variable to optimize is part of a causal model where a sequence of interventions can be performed . However , CBO does not account for the system ’ s temporal evolution thus breaking the time dependency structure existing among variables ( Fig . 1b ) . This will lead to sub-optimal solutions , especially in non-stationary scenarios . The same would happen when using Adaptive Bayesian Optimization [ ABO , 25 ] ( Fig . 1c ) or BO ( Fig . 1d ) . ABO captures the time dependency of the objective function but neither considers the causal structure among inputs nor their temporal evolution . BO disregards both the temporal and the causal structure . Our setting differs from both reinforcement learning ( RL ) and the multi-armed bandits setting ( MAB ) . Differently from MAB we consider interventions on continuous variables where the dynamic target variable has a non-stationary interventional distribution . In addition , compared to RL , we do not model the state dynamics explicitly and allow the agent to perform a number of explorative interventions which do not change the underlying state of the system , before selecting the optimal action . We discuss these points further in §1.1 . Dynamic Causal Bayesian Optimization2 , henceforth DCBO , accounts for both the causal relationships among input variables and the causality between inputs and outputs which might evolve over time . DCBO integrates CBO with dynamic Bayesian networks ( DBN ) , offering a novel approach for decision making under uncertainty within dynamical systems . DBN [ 19 ] are commonly used in time-series modelling and carry dependence assumptions that do not imply causation . Instead , in probabilistic causal models [ 27 ] , which form the basis for the CBO framework , graphs are buildt around causal information and allow us to reason about the effects of different interventions . By combining CBO with DBNs , the proposed methodology finds an optimal sequence of interventions which accounts for the causal temporal dynamics of the system . In addition , DCBO takes into account past optimal interventions and transfers this information across time , thus identifying the optimal intervention faster than competing approaches and at a lower cost . We make the following contributions : • We formulate a new class of optimization problems called Dynamic Causal Global Optimization ( DCGO ) where the objective functions account for the temporal causal dynamics among variables . • We give theoretical results demonstrating how interventional information can be transferred across time-steps depending on the topology of the causal graph . • Exploiting our theoretical results , we solve the optimization problem with DCBO . At every time step , DCBO constructs surrogate models for different intervention sets by integrating various sources of data while accounting for past interventions . • We analyze DCBO performance in a variety of settings comparing against CBO , ABO and BO . 1.1 Related Work . Dynamic Optimization Optimization in dynamic environments has been studied in the context of evolutionary algorithms [ 14 , 16 ] . More recently , other optimization techniques [ 28 , 32 , 10 ] have been adapted to dynamic settings , see e.g . [ 9 ] for a review . Focusing on BO , the literature on dynamic settings [ 3 , 7 , 25 ] is limited . The dynamic BO framework closest to this work is given by Nyikosa et al . [ 25 ] and focuses on functions defined on continuous spaces that follow a more complex behaviour than a simple Markov model . ABO treats the inputs as fixed and not as random variables , thereby disregarding their temporal evolution and , more importantly , breaking their causal dependencies . Causal Optimization Causal BO [ CBO , 2 ] focuses instead on the causal aspect of optimization and solves the problem of finding an optimal intervention in a DAG by modelling the intervention functions with single GPs or a multi-task GP model [ 1 ] . CBO disregards the existence of a temporal evolution in both the inputs and the output variable , treating them as i.i.d . overtime . While disregarding time significantly simplifies the problem , it prevents the identification of an optimal intervention at every t. Bandits and RL In the broader decision-making literature , causal relationships have been considered in the context of bandits [ 4 , 20–22 ] and reinforcement learning [ 23 , 8 , 13 , 36 , 24 ] . In these cases , actions or arms , correspond to interventions on a causal graph where there exists complex relationships between the agent ’ s decisions and the received rewards . While dynamic settings have been considered in acausal bandit algorithms [ 5 , 33 , 35 ] , causal MAB have focused on static settings . Dynamic settings are instead considered by RL algorithms and formalized through Markov decision processes ( MDP ) . 2A Python implementation is available at : https : //github.com/neildhir/DCBO . In the current formulation , DCBO does not consider an MDP as we do not have a notion of state and therefore do not require an explicit model of its dynamics . The system is fully specified by the causal model . As in BO , we focus on identifying a set of time-indexed optimal actions rather than an optimal policy . We allow the agent to perform explorative interventions that do not lead to state transitions . More importantly , differently from both MAB and RL , we allow for the integration of both observational and interventional data . An expanded discussion on the reason why DCBO should be used and the links between DCBO , CBO , ABO and RL is included in the supplement ( §8 ) . 2 Background and Problem Statement . Let random variables and values be denoted by upper-case and lower-case letters respectively . Vectors are represented shown in bold . do ( X = x ) represents an intervention on X whose value is set to x. p ( Y | X = x ) represents an observational distribution and p ( Y | do ( X = x ) ) represents an interventional distribution . This is the distribution of Y obtained by intervening on X and fixing its value to x in the data generating mechanism ( see Fig . 2 ) , irrespective of the values of its parents . Evaluating p ( Y | do ( X = x ) ) requires “ real ” interventions while p ( Y | X = x ) only requires “ observing ” the system . DO and DI denote observational and interventional datasets respectively . Consider a structural causal model ( SCM ) defined in Definition 1 . Definition 1 . ( Structural Causal Model ) [ 27 , p. 203 ] . A structural causal model M is a triple 〈U , V , F ) 〉 where U is a set of background variables ( also called exogenous ) , that are determined by factors outside of the model . V is a set { V1 , V2 , . . . , V|V| } of observable variables ( also called endogenous ) , that are determined by variables in the model ( i.e. , determined by variables in U ∪V ) . F is a set of functions { f1 , f2 , . . . , fn } such that each fi is a mapping from the respective domains of Ui ∪ Pa ( Vi ) to Vi , where Ui ⊆ U and Pa ( Vi ) ⊆ V \ Vi and the entire set F forms a mapping from U to V. In other words , each { fi ∈ vi ← fi ( Pa ( vi ) , ui ) | i = 1 , . . . , n } , assigns a value to Vi that depends on the values of the select set of variables ( Ui ∪ Pa ( Vi ) ) . M is associated to a directed acyclic graph ( DAG ) G , in which each node corresponds to a variable and the directed edges point from members of Pa ( Vi ) and Ui to Vi . We assume G to be known and leave the integration with causal discovery [ 15 ] methods for future work . Within V , we distinguish between three different types of variables : non-manipulative variables C , which can not be modified , treatment variables X that can be set to specific values and output variable Y that represents the agent ’ s outcome of interest . Exploiting the rules of do-calculus [ 27 ] one can compute p ( Y | do ( X = x ) ) using observational data . This often involves evaluating intractable integrals which can be approximated by using observational data to get a Monte Carlo estimate p̂ ( Y | do ( X = x ) ) ≈ p ( Y | do ( X = x ) ) . These approximations will be consistent when the number of samples drawn from p ( V ) is large . Causality in time One can encode the existence of causal mechanisms across time steps by explicitly representing these relationships with edges in an extended graph denoted by G0 : T . For instance , the DAG in Fig . 1 ( a ) can be seen as one of the DAGs in Fig . 1 ( b ) propagated in time . The DAG in Fig . 1 ( a ) captures both the causal structure existing across time steps and the causal mechanism within every “ time-slice ” t [ 19 ] . In order to reason about interventions that are implemented in a sequential manner , that is at time t we decide which intervention to perform in the system and so define : Definition 2 . Mt is the SCM at time step t defined asMt = 〈U0 : t , V0 : t , F0 : t〉 where 0 : t denotes the union of the corresponding variables or functions up to time t ( see Fig . 2 ) . V0 : t includes X0 : t = Xt , Y0 : t = Yt and C0 : t = Ct ∪C0 : t−1 . The functions in F0 : t corresponding to intervened variables are replaced by constant values while the exogenous variables related to them are excluded from U0 : t. Definition 3 . Gt is the causal graph associated to Mt . In Gt , the incoming edges in variables intervened at 0 : t− 1 are mutilated while intervened variables are represented by deterministic nodes ( squares ) – see Fig . 2 . Dynamic Causal Global Optimization ( DCGO ) The goal of this work is to find a sequence of interventions , optimizing a target variable , at each time step , in a causal DAG . Given Gt and Mt , at every time step t , we wish to optimize Yt by intervening on a subset of the manipulative variables Xt . The optimal intervention variables X ? s , t and intervention levels x ? s , t are given by : X ? s , t , x ? s , t = argmin Xs , t∈P ( Xt ) , xs∈D ( Xs , t ) E [ Yt | do ( Xs , t = xs , t ) ,1t > 0 · I0 : t−1 ] ( 1 ) where I0 : t−1 = ⋃t−1 i=0 do ( X ? s , i = x ? s , i ) denotes previous interventions , 1t > 0 is the indicator function and P ( Xt ) is the power set of Xt . D ( Xs , t ) represents the interventional domain of Xs , t . In the sequel we denote the previously intervened variables by IV0 : t−1 = ⋃t−1 i=0 X ? s , i and implemented intervention levels by IL0 : t−1 = ⋃t−1 i=0 x ? s , i . The cost of each intervention is given by cost ( Xs , t , xs , t ) . In order to solve the problem in Eq . ( 1 ) we make the following assumptions : Assumptions 1 . Denote by G ( t ) the causal graph including variables at time t in G0 : T and let Y PTt = Pa ( Yt ) ∩ Y0 : t−1 be the set of variables in G0 : T that are both parents of Yt and targets at previous time step . Let the set Y PNTt = Pa ( Yt ) \Y PTt be the complement and denote by fYt ( · ) the functional mapping for Yt in Mt . We make the following assumptions : 1 . Invariance of causal structure : G ( t ) = G ( 0 ) , ∀t > 0 . 2 . Additivity of fYt ( · ) that is Yt = fYt ( Pa ( Yt ) ) + with fYt ( Pa ( Yt ) ) = fYY ( Y PTt ) + fNYY ( Y PNTt ) where fYY and f NY Y are two generic unknown functions and ∼ N ( 0 , σ2 ) . 3 . Absence of unobserved confounders in G0 : T . Assumption ( 3 ) implies the absence of unobserved confounders at every time step . For instance , this is the case in Fig . 1a . Still in this DAG , Assumption ( 2 ) implies fYt ( Pa ( Yt ) ) = f Y Y ( Yt−1 ) + fNYY ( Zt ) + Yt , ∀t > 0 . Finally , Assumption ( 1 ) implies the existence of the same variables at every time step and a constant orientation of the edges among them for t > 0 . Notice that Assumptions 1 imply invariance of the causal structure within each time-slice , i.e . the structure , edges and vertices , concerning the nodes with the same time index . This means that , across time steps , both the graph and the functional relationships can change . Therefore , not only can the causal effects change significantly across time steps , but also the input dimensionality of the causal functions we model , might change . For instance , in the DAG of Fig . 3 ( c ) , the target function for Y2 has dimensionality 3 and a function fYt ( · ) that is completely different from the one assumed for Y1 that has only two parents . We can thus model a wide variety of settings and causal effects despite this assumption . Furthermore , even though we assume an additive structure for the functional relationship on Y , the use of GPs allow us to have flexible models with highly non-linear causal effects across different graph structures . In the causality literature , GP models are well established and have shown good performances compared to parametric linear and non-linear models ( see e.g . [ 31 , 34 , 37 ] ) . The sum of GPs gives a flexible and computationally tractable model that can be used to capture highly non-linear causal effects while helping with interpretability [ 12 , 11 ] .

Summary: In this paper, the authors extend the notion of casual Bayesian optimisation from a static to a dynamic scenario which, indeed, has many relevant applications. The paper theoretically analyses the proposed approach and run the algorithm in a set of real-world and synthetic data. This work is really interesting but I have some questions related to scalability and applicability that I hope the authors could help me understand.