Dynamic Causal Bayesian Optimization
Authors: Virginia Aglietti
Neil Dhir
Javier Gonzalez
Theo Damoulas
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian Optimization
Gaussian Processes
Causality

Abstract: We study the problem of performing a sequence of optimal interventions in a dynamic causal system where both the target variable of interest, and the inputs, evolve over time. This problem arises in a variety of domains including healthcare, operational research and policy design. Our approach, which we call Dynamic Causal Bayesian Optimisation (DCBO), brings together ideas from decision making, causal inference and Gaussian process (GP) emulation. DCBO is useful in scenarios where the causal effects are changing over time. Indeed, at every time step, DCBO identifies a local optimal intervention by integrating both observational and past interventional data collected from the system. We give theoretical results detailing how one can transfer interventional information across time steps and define a dynamic causal GP model which can be used to find optimal interventions in practice. Finally, we demonstrate how DCBO identifies optimal interventions faster than competing approaches in multiple settings and applications.

Dynamic Causal Bayesian Optimization
Virginia Aglietti" University of Warwick The Alan Turing Institute
Neil Dhir" The Alan Turing Institute ndhir@turing. ac uk
V. Aggiettiiwwrrwcck.kkkkk ac.
Javier González
Theodoros Damoulas University of Warwick
Microsoft Research Cambridge
Gonzalez. Javi ievier@ii armmiccoooofoe com
The Alan Turing Institute
Daamoullstwwarwickk ac.
Abstract
This paper studies the problem of performing sequence of optimal interventions in a causal dynamical system where both the target variable of interest and the inputs evolve over time. This problem arises in variety of domains e.g. system biology and operational research. Dynamic Causal Bayesian Optimization (DCBO) brings together ideas from sequential decision making, causal inference and Gaussian process (GP) emulation. DCBO useful in scenarios where all causal effects in graph changing over time. At every time step DCBO identifies local optimal intervention by integrating both observational and past interventional data collected from the system. We give theoretical results detailing how transfer interventional information across time steps and define a dynamic causal model which can be used quantify uncertainty and find optimal interventions in practice. We demonstrate how DCBO identifies optimal interventions faster than competing approaches in multiple settings and applications.
Introduction
Cansal Dimcmssio
Solving decision making problems in a variety of domains requires understanding of cause-effect relationships in system. This can be obtained by experimentation. However, deciding how to intervene at every point in time is particularly complex dynamical systems, due to the evolving nature of causal effects. For instance, companies need to decide how allocate scarce resources across different quarters. In system biology, scientists need to identify genes to knockout at specific points in time. This paper describes a probabilistic framework that finds such optimal interventions
(A)
DCBO
(c)
ABO
fileX,,
SiX
Tempoprl Dimersioon
CBO
(4)
BO
Xub
150
time.
1.0
3%
760
0%
Focusing on a specific example, consider setting which Y; denotes the unemployment-rate of an economy at time t, Z, is the economic growth and X, the inflation rate. Fig. [ depicts the causal graph [26] representing agent's understanding of causal links between these variables. The agent aims to determine, at each time step {0, 1,2). the optimal action to perform in order to minimize the current unemployment rate
Figure 1: DAG representation of dynamic causal global optimisation (DCGO) problem (a) and the DAG considered when using CBO, ABO BO address the same prob- lem. Shaded nodes gives observed variables while the arrows represent causal effects.
Denotes equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Y, while accounting for the intervention cost. The investigator could frame this setting as sequence of global optimization problems and find the solutions by resorting to Causal Bayesian Optimization [C. CBO extends Bayesian Optimization [BO, 30 to cases in which the variable to optimize is part of causal model where a sequence of interventions can be performed. However, CBO does not account for the system's 's temporal evolution thus breaking the time dependency structure existing among variables (Fig. [D). This will lead to sub-optimal solutions, especially in non-stationary scenarios. The same would happen when using Adaptive Bayesian Optimization [ABO, 225] (Fig. or BO (Fig. Tt), ABO captures the time dependency of the objective function but neither considers the causal structure among inputs their temporal evolution. BO disregards both the temporal and the causal structure. Our setting differs from both reinforcement learning (RL) and the multi-armed bandits setting (MAB). Differently from MAB consider interventions on continuous variables where the dynamic target variable has a non-stationary interventional distribution. In addition, compared to RI. we do not model the state dynamies explicitly and allow the agent to perform number of explorative interventions which do not change the underlying state of the system, before selecting the optimal action. We discuss these points further in 1
Dynamic Causal Bayesian Optimization henceforth DCBO, accounts for both the causal relationships among input variables and the causality between inputs and outputs which might evolve over time. DCBO integrates CBO with dynamic Bayesian networks (DBN), offering a novel approach for decision making under uncertainty within dynamical systems. DBN [9] are commonly used in time-series modelling and carry dependence assumptions that do not imply causation. Instead, in probabilistic causal models (27]. which form the basis for the CBO framework, graphs are buildt around causal information and allow us to reason about the effects of different interventions. By combining CBO with DBNS, the proposed methodology finds optimal sequence of interventions which accounts for the causal temporal dynamics of the system. In addition, DCBO takes into account past optimal interventions and transfers this information across time, thus identifying the optimal intervention faster than competing approaches and at a lower cost. We make the following contributions:
We formulate new class of optimization problems called Dynamic Causal Global Optimization (DCGO) where the objective functions account for the temporal causal dynamics among variables. We give theoretical results demonstrating how interventional information can be transferred across time-steps depending on the topology of the causal graph.
Exploiting our theoretical results, we solve the optimization problem with DCBO, At every time step. DCBO constructs surrogate models for different intervention sets by integrating various sources of data while accounting for past interventions.
We analyze DCBO performance in variety of settings comparing against CBO, ABO and BO.
1.1 Related Work
Dynamic Optimization Optimization in dynamic environments has been studied in the context of
evolutionary algorithms [6]. More recently, other optimization techniques (28 32 10 have been adapted to dynamic settings, see e.g. [9] for a review. Focusing on BO, the literature on dynamic
settings B 25 is limited. The dynamic BO framework closest to this work is given by Nyikosa et al.
(25) and focuses on functions defined on continuous spaces that follow more complex behaviour than a simple Markov model. ABO treats the inputs as fixed and not as random variables, thereby disregarding their temporal evolution and, more importantly, breaking their causal dependencies.
Causal Optimization Causal BO [CBO. 2 focuses instead on the causal aspect of optimization and solves the problem of finding an optimal intervention in a DAG by modelling the intervention functions with single GPs multi-task GP model M.. CBO disregards the existence of temporal evolution in both the inputs and the output variable, treating them as i.i.d. overtime. While disregarding time significantly simplifies the problem, it prevents the identification of optimal intervention every I.
Bandits and RL In the broader decision decision-making literature, causal relationships have been considered in the context of bandits [4. 22 and reinforcement learning (23) 8 13 36 24]. In these cases, actions or arms, correspond to interventions on a causal graph where there exists complex relationships between the agent's decisions and the received rewards. While dynamic settings have been considered in acausal bandit algorithms S 333 351, causal MAB have focused on static settings. Dynamic settings are instead considered by RL algorithms and formalized through Markov decision processes (MDP).
A Python implementation is available https: http::://ttttbbbeu http:///thu...ooeeeee///i//////a

---

In the current formulation, DCBO does not consider an MDP as we do not have notion of state and therefore do not require an explicit model of its dynamics. The system is fully specified by the causal model, As in BO, we focus on identifying set of time-indexed optimal actions rather than an optimal policy. We allow the agent to perform explorative interventions that do not lead to state transitions. More importantly, differently from both MAB and RI., we allow for the integration of both observational and interventional data. An expanded discussion on the reason why DCBO should be used and the links between DCBO, CBO, ABO and RL is included in the supplement ($8).
2 Background and Problem Statement
Let random variables and values be denoted by upper-case and lower-case letters respectively, Vectors are represented shown in bold. do = z) represents an intervention on X whose value is set to r. p(Y | X .²) represents an observational distribution and p(Y') | do(X interventional distribution. This is the distribution of Y obtained by intervening on X and fixing its value to x in the data generating mechanism (see Fig. 2). irrespective of the values of its parents.
x)) represents an
Evaluating p(Y | do(X = r)) requires "real" interventions while p(Y X = x) only requires "observing" the system. DG and D' denote observational and interventional datasets respectively, Consider structural causal model (SCM) defined in Definition [
Definition 1. (Structural Causal Model) 277 p. 203]. A structural causal model M is a triple (U, V,F)) where U is a set of background variables (also called exogenous), that are determined by factors outside of the model. V is a set (V1. V22 .... V of observable variables (also called endogenous), that are determined by variables in the model (i.e., determined by variables in U UUW UU.. F is set of functions {fi. ........] such that each fa a mapping from the respective domains of U; U Pa(V.) to V,, where U, C U and Pa(V:) C V Vi and the entire set F forms a mapping from U to V. other words, each (fi E o < (Pa(())) (4) | ii= 11=1,-. 1,..., ..). assigns a value to V, that depends on the values of the select set of variables (U; U (V()))
M is associated to a directed acyclic graph (DAG) G, in which each node corresponds to variable and the directed edges point from members of Pa(V, V.) and U, to V4. We assume G to be known and leave the integration with causal discovery [15] methods for future work. Within V, we distinguish between three different types of variables: non-manipulatiie variables C, which cannot be modified, treatment variables X that can be set specific values and output variable Y that represents the agent's outcome of interest. Exploiting the rules of do-calculus (27) one can compute p(Y do(X = =)) using observational data. This often involves evaluating intractable integrals which can be approximated by using observational data to get a Monte Carlo estimate P(Y | do(X = x)) so p(Y do = x)).
These approximations will be consistent when the number of samples drawn from p(V) is large.
Causality in time One can encode the existence of causal mechanisms across time steps by explicitly representing these relationships with edges in extended graph denoted by Go.r. For instance, the DAG in Fig. aa) can be seen as one of the DAGS in Fig. [bb) propagated in time. The DAG in Fig. I () captures both the causal structure existing across time steps and the causal mechanism within every "time-slice" t [19]. In order to reason about interventions that are implemented in a sequential manner, that is at time t we decide which intervention to perform in the system and so define:
Definition 2. M, is the SCM at time step t defined as M,- = (Uo.., V Vo.,, Fo.t) where 0: f denotes the union of the corresponding variables or functions up to time t (see Fig. 2). Vo. includes Xu = X,, Yu. Y, and Co. C U Cua--1- The functions in Fu., corresponding to intervened variables are replaced by constant values while the exogenous variables related to them are excluded from U..-
Definition 3. G, is the causal graph associated to Mc. In Gr. the incoming edges in variables intervened at : are mutilated while intervened variables are represented by deterministic nodes (squares) see Fig. 2
Dynamic Causal Global Optimization (DCGO) The goal of this work is to find a sequence of interventions, optimizing a target variable, at each time step, in a causal DAG. Given Ge and Mc, at every time step f, we wish optimize Y, by intervening on subset of the manipulative variables X,. The optimal intervention variables x and intervention levels Xx, are given by:
X,,xx
mig X EPX.),,,,)
E|G do(X,, = xs,)) le Ioa-1]]
(1)
3

---

DAG
X, = (x,(ex))
Fo Ju.
C== 1
XoC
Zo = (2, 62)
U.- (ex; (X)
Yo Yo
Y, fn (Y.
X. (Xo. Zo}
Gi-0
70
(Mn, no)
ung E(Vo do x.n)) X. k.
6%
Yo
6Xa O ex,
No = fx,(ex.)
Foc1 (fx, fr, fys - An) Cpu (ot.Yo)
Xo
X1
2, 122(1
Uol (Xx (2 You - V/
12)
Y, = fy (am.%)) 6x))
Xod (x1,2)
G-11
X1 fx, ex,)
O
21 = fa (30, X(.e2)
1= (X ang min E(Yi do((..,) x.,)) fo]
1o
Vi
1, fv (Yo, ei,en)
6Xa
EX2
X₂ fxp(sx,)
Foz (xx.. Mar Bar Pys, Bar fx, fen Ix Jen
Xo
2: f2,()
Uo2 (ex.: (%: E2 (2a (1))
%
Y:- (x0))%)
Xu: (N2.75
Sh-2
X, fky ( ma
Cuz = (Xo, Yo, 21.
2. fz, (Roy (L)
You 1/5
Y. Fr. (Yo, 22,9))
1/6
X; (R (xL))
Ia (Ze,
ang mnin E(Ys do(X,, = 4. X. Xah. X,GUXXX.a)
Vi
2. (x.
1/2 (Y, Zz.ro))
Figure 2: Structural equation models considered by DCBO at every time step E {0, 2, Exogenous noise variables e; are depicted here but are omitted in the remainder the paper, to avoid clutter. For every t, G. is mutilated version of Gi-1 reflecting the optimal intervention implemented in the system at t - 1 which are represented by squares. The SCM functions in F Fox corresponding to the intervened variables are set constant values. The exogenous variables that only related to the intervened variables are excluded from U,. Co. is given by the set {C, CCCUCoa-UUYoo
where To:-1- = - do x2.)) denotes previous interventions, 1 1s- is the indicator function and PXX,) is the power set of X, D(X,,t) represents the interventional domain of X,,-- In the sequel we denote the previously intervened variables by T U x and implemented intervention levels by T U x The cost of each intervention is given by cost bost(X,,t+ Xs.t).
In order to solve the problem in Eq. [T) we make the following assumptions :
Assumptions 1. Denote by G(t) the causal graph including variables time t in Go.T and let YPT Pa(Y.) n Yo.-1 be the set of variables in Go:r that are both parents of Y, and targets at previous time step. Let the set Y,PNT Pa(Y,) \Y.'T be the complement and denote by fy, (.) the functional mapping for Y, in My. We make the following assumptions:
1. Invariance of causal structure: ((() = G()) 4 0.
2. Additivity of (-) that is Y, = fr, (Pa(Y)) + e with fr. (Pa(Yi)) = F. + (YPNT) where f3 and JNY are two generic unknown functions and e ((
3. Absence of unobserved confounders in Go:r-
Assumption (3) implies the absence of unobserved confounders at every time step. For instance, this is the case in Fig. [a. Still in this DAG, Assumption (2) implics fy, (Pa(Y)) = ff (Y--))+ (XY(Z,) + Ey,, Vt > O. Finally, Assumption (1) implies the existence of the same variables at every time step and a constant orientation of the edges among them for > 0.
Notice that Assumptions liplyy invariance of the causal structure within each time-slice, i.e. the structure, edges and vertices, concerning the nodes with the same time index. This means that, across time steps, both the graph and the functional relationships can change. Therefore, not only can the causal effects change significantly across time steps, but also the input dimensionality of the causal functions we model, might change. For instance, in the DAG of Fig. c).. the target function for Y2 has dimensionality and a function fr, (-) that completely different from the one assumed for Y, that has only two parents. We can thus model wide variety of settings and causal effeets despite this

---

assumption. Furthermore, even though we assume an additive structure for the functional relationship on Y. the use of GPs allow us have flexible models with highly non-linear causal effects across different graph structures. In the causality literature, GP models are well established and have shown good performances compared to parametric linear and non-linear models (see e.g. [31 (7)) ). The sum of GPs gives flexible and computationally tractable model that can be used to capture highly non-linear causal effects while helping with interpretability [12]
3 Methodology
In this section we introduce Dynamic Causal Bayesian Optimization (DCBO), a novel methodology addressing the problem in Eq. (T). We first study the correlation among objective functions for two consecutive time steps and use it to derive a recursion formula that, based on the topology of the graph, expresses the causal effects at time t function of previously implemented interventions (see square nodes in Fig. 2 Exploiting these results, we develop a new surrogate model for the objective functions that can be used within CBO framework find the optimal sequence of interventions. This model enables the integration of observational data and interventional data, collected at previous time-steps and interventional data collected at time t, thereby accelerating the identification of the current optimal intervention.
3.1 Characterization of the time structure in a DAG with time dependent variables
The following result provides a theoretical foundation for the dynamic causal GP model introduced later. In particular, derives a recursion formula allowing us to express the objective function at time t as function of the objective functions corresponding to the optimal interventions at previous time steps. The proof given in the appendix ($2).
Definition 4. Consider DAG Go.r and the objective function E[Y; do(X,,t = =x,e)= A0a for a generic time step E {o....,7) T... Denote by Y,PT = (Pa(Y,) n Yo:-1) the parents of Y, that targets previous time steps and by Y Pa(Y,) VYPT the remaining parents. For any
X.,t E P(X.) and 1 S Xox--1 we define the following sets:
XPY Xx, (a(r) includes the variables in Xs, that are parents of Y,.
rBY = 1 n Pa(Y)) includes the variables in I that are parents of Y,
W C Pa(Y.) such that Pa(Yi) (Pa(Y)) Yo.-1) U X U 1 U. W includes variables that are parents of Y, but are not targets nor intervened variables.
The values of 1o.-11 X 1 and W will be denoted by i, xx", PY and w respectively.
Theorem 1. Time operator. Consider a DAG GO.T and the related SCM satisfying Assump-
tions It is possible to prove that, vX,,1 E P(Xi), the intervention function fe,((x) = B[Y, do X,,1 =x), 11>0 Ioa-1] with fx,((x) D(X,,)) + R can be written as:
(x) = f (f*) + Bpp))=B((wddxxx..... (p NY (x PY ,PY w)]
where f== (T=(B[>) do (X2,, = xx;,) Toa--llxxey that is the set of previously observed optimal
targets that are parents of Ye. fL denotes the function mapping y.PT to Y, and lor represents the function mapping Y, PNT to Y..
Eq. 2 reduces to [p(w] da( -x). [FNY [f (xPY, PY,, []]] when Y, does not depend on previous targets. This the setting considered CBO that can be thus seen as a particular instance of DCBO. Exploiting Assumptions (I). it is possible to further expand the second term in Eq. 2 to get the following expression. proof is given in the supplement ($
Corollary 1. Given Assumptions [ we can write:
Ep(w/ xx)x,.=x))i) [ (x PY PP w)] Epc... (x PP
iPY, [(C(W))Wew) (3)
where p(Uo.) is the distribution for the exogenous variables up time and C(W) is given by: if R = Ø R X., U TV
[uw XPW. PW 1PW)
C(W)
(uw, Pw jPW r)
if
fw (uw, x ipw C(R))
if R g X U U
5

---

where fw represents the functional happing for W in the SCM and uw is the set of exogenous variables with edges into W. xPW and are the values corresponding to X and 1a-1 which in turn represent the subset of variables x,, and f that are parents of W. Finally r is the value of R Pa(W) (X U JPW
Examples for Eq. (2): For the DAG in Fig. 1 [a), at time = 1 and with 1
[Zo}. we
have E[y do 1 2) fo] f (vā) + (2). Indeed in this case W Ø, x and
f* = Luo E|Yo| do Zo =0]]}. Still at t = 1 and with 1 (Zo}. the objective function
for X (X1) be written ((6) Pataal =d) [ (=1) as W {Z1). All
derivations for these expressions and alternative graphs are given in the supplement
3.2 Restricting the search space
The search space for the problem in Eq. grows exponentially with X,] thus slowing down the identification of the optimal intervention when G includes more than a few nodes. Indeed, a naive approach of finding X f = ... would be to explore the 2 sets in P(X,) at every and keep 2 x models for the objective functions. In the static setting, CBO reduces the search space by exploiting the results in [2T]. In particular, identifies a subset of variables M C PPX)) worth intervening on thus reducing the size of the exploration set to M
In our dynamic setting, the objective functions change at every time step depending on the previously implemented interventions and one would need recompute MI at every l. However, it is possible to show that, given Assumptions the search space remains constant over time. Denote by M, the set M at time i and let Mi represent the set at 0 which corresponds to M computed in CBO. For t > 0 it is possible to prove that:
Proposition 3.1. MIS in time. If Assumptions are satisfied, M, = Ml for t 0.
3.3 Dynamic Causal GP model
Here introduce the Dynamic Causal GP model that is used surrogate model for the objective functions in Eq. [T. The prior parameters are constructed by exploiting the recursion in Eq. (2). At each time step the agent explores the sets in M, S P(X,) by selecting the next intervention to be the one maximizing given acquisition function. The DCBO algorithm is shown in Algorithm [
At each time step and for each X,,t E Ml, we place a GP prior the objective function (x) = E[y do(X, (x,,, = x). 1220 Io We construct the prior parameters
Prior Surrogate Model
exploiting the recursive expression in Eq. 2
[x, ~ GP (x), (x,, where
(x,(()) = E [ (1*) - (x PY, iPY w]]] (x, x') = kiesr xx x') 0 (x)o with
(x) = VIFY (f+) + É (xPY jP,, w)]
and kir=r(x, x²) = exp( ||x ||x-x'11 x'1 21
represents the radial basis function kernel 29
We have it that
E [ (xPY iPY w)] Ep(w =x).) [ (xPY PY w)]
represents the expected value of ) JPY w) with respect to p(w do(X..t = x) i)) which is estimated via the do-calculus using observational data. The outer expectation in (x,((x) and the variance in ox,((x) are computed with respect to KJJYJJN which is also estimated using observational data. In this work model fy, Jy and all functions in the SCM by independent GPs.
Both (x) and (x) can be equivalently written by exploiting the equivalence in Eq. (3). In both cases, this prior construction allows the integration of three different types of data: observational data, interventional data collected at time and the optimal interventional data points collected in the past. The former is used to estimate the SCM model and p(w do(X,, x) i) via the rules of

---

do-calculus. The optimal interventional data points at 0 :t 1 determine the shift f (f*) while the interventional data collected at time are used update the prior distribution on (x,) Similar prior constructions were previously considered in static settings [ where only observational and interventional data at the current time step were used. The additional shift term appears here as there exists causal dynamics in the target variables and the objective function is affected by previous decisions. Fig. in the appendix shows a synthetic example in which accounting for the dynamic aspect in the prior formulation leads to more accurate GP posterior compared to the baselines, especially when the the optimum location changes across time steps.
Likelihood Let a.t (X', Y!,) be the set of interventional data points collected for X with X being vector of intervention values and Y!, representing the corresponding vector of observed target values. As in stan-
Algorithm 1: DCBO
Data: D0,
|b,,)) Go.r, H.
Result: Optimal intervention path
( )==
Initialise: M, Dj and initial optimal DJ Ø. for
dard BO we assume each Va.t in Y 8.0 to be
0. ......''
noisy observation of the function (x) that is (x, = (x) + Ea,t with 6a,2 ~ (O0, o²) for s E {1..... [M]] } and
Initialise dynamic causal models for all
X.. E E,, using Di,r1 if
2. Initialise interventional dataset (DL,theeoo.... M.l)
(o.....T) T.. In compact form, the joint likelihood function for D',, is p( 5, fe.e, o²) (Nf.., II...
for -....... do
1. Compute Blg,(()) for each X., My.
2. Obtain (s", a²)
Acquisition Function Given our surrogate models at time f, the agent selects the interven- tions to implement by solving a Causal Bayesian Optimization problem (2. The agent explores the sets in M, and decides where to intervene by maximizing the Causal Expected Improvement
3. Intervene and augment D
4. Update posterior for fa=st,
end
3. Return the optimal intervention (X;,,,
4. Append optimal interventional data
D D ((X),; xx,).
(EI). Denote by y the optimal observed target
end
value in { {Y) s.t /mml
[M.l that is the optimal observed target across all intervention sets at time The
Causal EI is given by
(x,,(() (max...) vt: 0)/ htt//csstttxc x...))
Let 01, [M] be solutions of the optimization of (x) for each set in M, and a* max. a.a.) ApM.)). The next best intervention to explore at time t is given by s* argmax.e (1..... Os. Therefore, the set-value pair to intervene on is (s*, a²)) At every I, the agent implement И explorative interventions in the system which are selected by maximizing the Causal EL. Once the budget I is exhausted, the agent implements what call the decision intervention 4,, that the optimal intervention found at the current time step, and move forward to a new optimization at t + carrying the information in V The parameter H determines the level of exploration of the system and acts as a budget for the CBO algorithm. Its value is determined by the agent and is generally problem specific.
Posterior Surrogate Model For any set X,,t E Me, the posterior distribution p(fe, D(,,) can be derived analytically via standard GP updates. p(fa,c D(,)) will also be GP with parameters
(x D',,) mx,:(x) + (x,c(x, X [Ko, (X',
xx) (X'
o'I((Y' mxx(X') and
(x,((x,' x' D(,.)= kx,((x,)) x') 1x, X (ks, (X. x') ) o (
4 Experiments
We evaluate the performance of DCBO in a variety of synthetic and real world settings with DAGS given in Fig. 3 We first run the algorithm for a stationary setting where both the graph structure and the SCM do not change over time (STAT.). We then consider scenario characterised by increased observation noise (NOISY) for the manipulative variables and a settings where observational data are missing at some time steps (MISS.). Still assuming stationarity, we then test the algorithm in a DAG where there are multivariate interventions in M., (MUL.TIV.). Finally, we run DCBO for non-stationary graph where both the SCM and the DAG change over time (NONSTAT.). To conclude,
7

---

we use DCBO to optimize the unmpployment rate of closed economy (DAG in Fig. 3d ECON.) and to find the optimal intervention in a system of ordinary differential equation modelling a real predator-prey system (DAG in Fig. 3e ODE). We provide discussion on the applicability of DCBO to real-world problems in of the supplement together with all implementation details.
Baselines We compare against the algorithms in Fig. I Note that, by constructions, ABO and Bo intervene on all manipulative variables while DCBO and CBO explore only M, every t. In addition, both DCBO and ABO reduce to CBO and BO at the first time step. We assume the availability of an observational dataset Do and set unit intervention cost for all variables.
Performance metric We run all experiments for 10 replicates and show the average convergence path at every time step. We then compute the values of modified "gap" metrid" across time steps and with standard errors across replicates. The metric is defined as
y (x(x)) (XXnitt
H (xx,))))) H
(111 H H11
G2 y* y (Xmit)
=
(4)
H
where v(+) represents the evaluation of the objective function, y² is the global minimum, and Xnit and x,,, are the first and best evaluated point, respectively. The term Hx with Hx.,,) denoting the number of explorative trials needed to reach x captures the speed of the optimization. This term equal to zero when the algorithm not converged and equal (H - 1) H when the algorithm converges at the first trial. We have 0 I G2 < 1 with higher values denoting better performances. For each method we also show the average percentage of replicates where the optimal intervention set X is identified.
Xa
X2
X2
Xo
X1
X2
Xo
X
X2
R
R
R2
Yo
V
Y22
Y
V12
Zo
21
21
To
T
W
W
W
Go
G
Za
7/2
%o
Yi
Uo
U1
U/2
(a) MULTIV.
(b) IND.
(c) NONSTAT.
(d) ECON.
(e) ODE.
Figure 3: DAGS used in the experimental sections for the real ($4.2) and synthetic data ($4.1
4.1 Synthetic Experiments
Stationary DAG and SCM (STAT.) We run the algorithms for the DAG in Fig. aa) with T' 3 and
N 10. For ? > 0, DCBO converges to the optimal value faster than competing approaches (see Fig. 2 in the supplement, right panel, 3rd row). DCBO identifies the optimal intervention set in 93% of the replicates (Table 2 and reaches the highest average gap metric (Table [T. In this experiment the location of the optimum changes significantly both in terms of optimal set and intervention value when going from t = 0 to f = 1. This information is incorporated by DCBO through the prior dependency on - In addition, ABO performance improves over time as it accumulates interventional data and uses them to fit the temporal dimension of the surrogate model. This benefits ABO in a stationary setting but might penalise it in non-stationary setting where the objective functions change significantly.
Noisy manipulative variables (NoISY): The benefit of using DCBO becomes more apparent when the manipulative variables observations are noisy while the evolution of the target variable is more accurately detected. In this case both the convergence of DCBO and CBO are slowed down by noisy observations which are diluting the information provided by the do-calculus making the priors less informative. However, the DCBO prior dependency on Vo allows it to correctly identify the shift in the target variable thus improving the prior accuracy and the speed-up up of the algorithm (Fig. 4.
Missing observational data (MISS.) Incorporating dynamic information in the surrogate model allows us to efficiently optimise a target variable even in setting where observational data are missing.
This metric is modified version of the one used in 18
8

---

[=2 DCBO
CRO
ABO
ddeX,, -x,]]
10
15
coat Na.
cost
coat x X,21
Figure 4: Experiment NOISY. Convergence DCBO and competing methods across replicates. The dashed black line --- gives the optimal outcome ye Vt. Shaded areas are + one standard deviation.
We consider the DAG in Fig. a with 7 6, N 10 for the first three time steps and afterwards. DCBO uses the observational distributions learned with data from the first three time steps to construct the prior for > 3. On the contrary, CBO uses the standard prior for t > 3. this setting DCBO consistently outperforms CBO every time step. However, ABO performance improves over time and outperforms DCBO starting from / = 4 due to its ability to exploit all interventional data collected over time (see Fig. in the supplement).
Multivariate intervention sets (MULTIV.) When the optimal intervention set is nultivariate. both DCBO and CBO convergence speed worsen. For instance, for the DAG in Fig. Bal M = 5 thus both CBO and DCBO will have to perform more explorative interventions before finding the optimum. At the same time, ABO and BO consider interventions only (W,, X2, Ze), Vt and need to explore even higher intervention space. The performance all methods decrease in this case (Table but DCBO still identifies the optimal intervention set in 93% of the replicates (Table
Independent manipulative variables (IND.): Having to explore multiple intervention sets signifi- cantly penalises DCBO and CBO when there is causal relationship among munipulative variables which are also the only parents intervention is {X,, Ze} every time step. In this case, exploring MI and propagating uncertainty in the causal prior slows down DCBO convergence and decreases both its performance (Table ! and capability to identify the optimal intervention set (Table 2
target. This the case for the DAG in Fig. 3ig where the optimal
Non-stationary DAG and SCM (NONSTAT.): DCBO outperforms all approackes non-stationary settings where both the DAG and the SCM change overtime see Fig, 3c Indeed, DCBO can timely incorporate changes in the system via the dynamic causal prior construction while CBO, BO and ABO need to perform several interventions before accurately learning the new objective functions.
Table 1:
Average G, across 10 replicates and time steps. See Fig. for summary of the baselines. Higher values are better. The best result for each experiment in bold. Standard errors in brackets.
Synthetic data
Real data
STAT.
MISS.
NOISY
MULTIV.
IND.
NONSTAT.
ECON.
ODE
0.88
0.84
0.75
0.49
0.48
0.69
0.64
0.67
DCBO
(0.00)
(0.01)
(0.00)
(0.01)
(0.04)
(0.00)
(0.01)
(0.00)
0.70
0.70
0.51
0.48
0.47
0.61
0.61
0.65
CBO
(0.01)
(0.02)
(0.02)
(0.09)
(0.07)
(0.00)
(0.01)
(0.00)
0.56
0.49
0.49
0.39
0.54
0.38
0.57
0.48
ABO
(0.01)
(0.02)
(0.04)
(0.21)
(0.01)
(0.02)
(0.02)
(0.01)
0.54
0.48
0.38
0.35
0.50
0.38
0.50
0.44
BO
(0.02)
(0.03)
(0.05)
(0.08)
(0.01)
(0.03)
(0.01)
(0.03)
4.2
Real experiments
Real-Word World Economic data (ECON.) We use DCBO to minimize the unemployment rate U, of closed economy. We consider its causal relationships with economic growth (G.), inflation rate (R₂) and fiscal policy (T, 4 Inspired by the economic example [17] we consider the DAG in
'The causality between economic variables is oversimplified in this example thus the results cannot be used guide public policy and only meant showcase how DCBO be used within real application.

---

Table 2: Average replicates across time steps for which X identified. See Fig. 1 for
summary of the baselines. Higher values are better. The best result for each experiment in bold.
Synthetic data
Real data
STAT.
MISS.
NOISY
MULTIV.
IND.
NONSTAT.
ECON.
ODE
DCBO
93.00
58.00
100.00
93.00
93.00
100.00
86.67
33.3
CBO
90.00
85.00
90.00
90.0
90.00
100.00
93.33
33.3
ABO
0.00
0.00
0.00
0.00
100.00
0.00
66.67
0.00
B0
0.00
0.00
0.00
0.00
100.00
0.00
66.67
0.00
Fig. 3d where RG and T; are considered manipulative variables we need to intervene on to minimize every time step. Time series data for countrie are used to construct non-pan non-parametric simulator and to compute the causal prior for both DCBO and CBO. DCBO convergences to the optimal the appendix). The optimal
log(U;
intervention faster than competing approaches (see Table I and Fig. sequence of interventions found in this experiment is equal to {(T). Ro (9.38, 2.000), (Ti, Ri) (0.53, 6.00), (7>) - (0.012)) which is consistent with domain knowledge.
Planktonic predator-prey community in chemostat (ODE) We investigate a biological system in which two species interact, one as a predator and the other as prey, with the goal of identifying the intervention reducing the concentration of dead animals in the chemostat see D, in Fig, [c] We use the system of ordinary differential equations (ODE) given by 66 our SCM and construct the DAG by rolling out the temporal variable dependencies in the ODE while removing graph cycles. Observational data are provided in and are use to compute the dynamic causal prior. DCBO outperforms competing methods in term of average gap metric and identifies the optimum faster (Table Dp. Additional details can be found in the supplement ($6)
Conclusions
We consider the problem of finding sequence of optimal interventions in a causal graph where causal temporal dependencies exist between variables. We propose the Dynamic Causal Bayesian Optimization (DCBO) algorithm which finds the optimal intervention every time step by intervening in the system according to a causal acquisition function. Importantly, for each possible intervention propose to
surrogate model that incorporates information from previous interventions implemented in the system. This constructed by exploiting theoretical results establishing the correlation structure among objective functions for two consecutive time steps function of the topology of the causal graph. We discuss the DCBO performance in variety of setting characterized by different DAG properties and stationarity assumptions. Future work will focus on extending our theoretical results to more general DAG structures thus allowing for unobserved confounders and a changing DAG topology within each time step. In addition, we will work on combining the proposed framework with causal discovery algorithm so as to account for uncertainty in the graph structure.
Acknowledgements
This work was supported by the EPSRC grant EP/L016710/1, The Alan Turing Institute under EPSRC grant EP/N5 19/1,, the Defence and Security Programme at The Alan Turing Institute, funded by the UK Government and the Lloyds Register Foundation programme on Data Centric Engineering through the London Air Quality project. TD acknowledges support from UKRI Turing AI Fellowship (EP/V02678X//I.
"Data were downloaded from sttps://www.dtt... org/ [Accessed: 01/04/2021). All details
in the supplement.
10

---

References
[1] Aglietti, V., Damoulas, T., Álvarez, M., and González, Multi-task causal learning with gaussian processes. In Advances in Neural Information Processing Systems, volume 33, pp. 6293-6304, 2020.
[2] Aglietti, Lu, X., Paleyes, A., and González, J. Causal Bayesian Optimization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1115-31644 PMLR, 26-28 Aug 2020.
[3] Azimi, J.. Jalali, A., and Fern, X. Dynamic batch bayesian optimization. arXiv preprint 11X.v:11 2011.
[4] Bareinboim, E., Forney, A., and Pearl, J. Bandits with unobserved confounders: A causal approach. Advances Neural Information Processing Systems, 28: 342-1350, 2015.
[5] Besbes, O., Gur, Y., and Zeevi, A. Stochastic multi-armeedbandii problem with non-stationary rewards. Advances in neural information processing systems, 27:199-207 2014.
[6] Blasius, B., Rudolf, L., Weithoff, G., Gaedke, U., and Fussmann, G. F. Long-term cyclic persistence in an experimental predator-prey system. Nature, 577(7789):226-236 2020.
[7] Bogunovic, I., Scarlett, J., and Cevher, V. Time-varying gaussian process bandit optimization. In Artificial Intelligence and Statistics, pp. 314-323. PMLR, 2016.
[8] Buesing, I.., Weber, T., Zwols, Y., Racaniere, S., Guez, A., Lespiau, J.-B., and Heess, N. Woulda, coulda, shoulda: Counterfactuulll--uuiee policy search. arXiv preprint 1rXxv:8811006277 2018.
[9] Cruz, C., González, J. R., and Pelta, D. A. Optimization in dynamic environments: survey on problems, methods and measures. Soft Computing, 1577):1427-148, 2011.
[10] De, M. K., Slawomir, J., and Mark, B. Stochastic diffusion search: Partial function evaluation in swarm intelligence dynamic optimisation. In Stigmergic optimization, pp. 185- 185-207. Springer, 2006.
[11] Duvenaud, D. Automatic model construction with Gaussian processes. PhD thesis, University of Cambridge, 2014.
[12] Duvenaud, D., Nickisch, H., and Rasmussen, C. E. Additive Gaussian Processes. arXiv preprint arXiv:1112.4394, 2011.
[13] Foerster, Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[14] Fogel, J., Owens, A. J., and Walsh, M. J. Artificial intelligence through simulated evolution. 1966.
[15] Glymour, C., Zhang, K., and Spirtes, P. Review of causal discovery methods based graphical models. Frontiers in Genetics, 10, 2019,
[16] Goldberg, D. F. and Smith, R. F. Nonstationary function optimization using genetic algorithms with dominance and diploidy. Genetic algorithms and their applications: proceedings of the second International Conference Genetic Algorithms: July 28-31, 1987 the Massachusetts Institute of Technology, Cambridge, MA. Hillsdale, NJ: L. Erlhaum Associates, 1987., 1987.
[17] Huang, B., Zhang, K., Gong, M., and Glymour, C. Causal discovery and forecasting in nonstationary environments with state-space models. In International Conference on Machine Learning, pp. 2901-2910. PMLR, 2019.
[18] Huang, D., Allen, T. T., Notz, I. and Zeng, N. Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of global optimization, 34(3):441 1-66 2006.
11

---

[19] Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques Adaptive Computation and Machine Learning. The MIT Press, 2009. ISBN 0262013193.
[20] Lattimore, F., Lattimore, T., and Reid, M. D. Causal bandits: Learning good interventions via causal inference. In Advances in Neural Information Processing Systems, pp. 1181-1189, 2016.
[21] Lee, S. and Bareinboim, E. Structural causal bandits: where intervene? Advances in Neural Information Processing Systems 31, 31, 2018
[22] Lee, S. and Bareinboim, E. Structural causal bandits with non-manipulable variables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4164-4172. 2019.
[23] Lu, C., Schölkopf. B., and Heráández-Lobato, J. M. Deconfounding reinforcement learning in observational settings. arXiv preprint arXiv: 1812. 10576, 2018.
[24] Madumal, P., Miller, T., Sonenberg, L., and Vetere, Explainable reinforcement learning through causal lens. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 2493-2500 2020.
[25] Nyikosa, M., Osborne, M. A., and Roberts, J. Bayesian optimization for dynamic problems, 2018.
[26] Pearl, Causal diagrams for empirical research. Biometrika, 82(4):669-683 1995.
[27] Pearl, Causality: models, reasoning and inference, volume 29, Springer, 2000.
[28] Pelta, D., Cruz, C., and Verdegay, J. Simple control rules in a cooperative system for dynamic optimisation problems. International Journal of General Systems, 38(7)7001-777, 2009.
[29] Rasmussen, C. E. Gaussian processes in machine learning. In Summer School Machine Learning, pp. 63-71. Springer, 2003.
[30] Shahriari, B., Swersky, K., Wang, 7., Adams, R. P., and De Freitas, N. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2015,
[31] Silva, R. and Gramacy, R. B. Gaussian Process Structural Equation Models with Latent Variables. arXiv preprint arXiv:1002.4802, 2010.
[32] Trojanowski. K. and Wierzchon, S. T. Immune-based algorithms for dynamic optimization. Information Sciences, (79(11):1495-11
2009.
[33] Villar, S. S., Bowden, J., and Wason, J. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Staristical science: review journal of the Institute of Mathematical Statistics, 30(2):199, 2015.
[34] Witty, S., Takatsu, K., Jensen, D., and Mansinghka, V. Causal Inference using Gaussian Processes with Structured Latent Confounders. In International Conference on Machine Learning, pp. 10313-10323, PMLR, 2020.
[35] Wu, Q. Iyer, N., and Wang, H. Learning contextual bandits in non-stationary environment. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 495-504, 2018.
[36] Zhang, J. and Bareinboim, E. Near-optimal reinforcement learning in dynamic treatment regimes. In Advances in Neural Information Processing Systems 32 pre-proceedings (NeurIPS), 2019.
[37] Zhang, K., Schölkopf, B., and Janzing, D. Invariant Gaussian Process Latent Variable Models and Application in Causal Discovery. arXiv preprint arXiv: arXiv:/203.33:0 1203.3534, 2012.
12


Reviews:
[{"review_id": "Z3KvNeCB1Is", "reviewer": "Reviewer_Rw3m", "summary": "This paper works with the Dynamic Causal Global Optimization problem. For a temporal structural causal model, which is a sequence of structural causal models with causality in time, meaning there are dependencies on the values of variables from previous step. The goal is to optimize the outcome variable by manipulating some observable variables (those are chosen by the model).\n\nAn algorithm called dynamic causal bayesian optimization is proposed to solve such a problem. The objective function $f_{s,t}$ is decomposed into the sum of a function depending on the optimized observed variables and another function depending on others (variables in the current time). Thus the optimization can be done step by step in time.\n\nA dynamic causal GP model is introduced for practically implement the DCBO algorithm.\n\nFor experiments, both Synthetic and Real data are used, in several different structural causal models, to compare the behavior of DCBO, to that of CBO, ABO and BO.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(I) The problem and methods are noval as far as I know. \n\n(II) The results are well supported. The methods used are appropriate. It is a complete work.\n\n(III) The paper is well organized, the structure is clear, important conclusions and concepts are provided in math environments (theorem / lemma / corollary / definition). All conclusions have formal proofs. However, there are some typos and font mismatches in the text and formulas.\n\n(IV) This work solves the dynamic problem and builds a relation to previous CBO problem. It is difficult for me to determine how important this work is, because of my lack of experience in this area.\n\n----------------------\nQuestions:\n- The conclusion depends on the assumptions. So:\n   - How likely the additivity of $f_{Y_t}$ is satisfied? What is the difficulty in solving the problem without additivity (I guess, the theorem 1 will no longer be of the same format, the sum of $f^Y$ and $f^{NY}$).\n   - What will happen if the structure of causal model changes in time? (It is assumed $G(t)=G(0)$ in the assumption).\n- The practical algorithm depends on Gaussian Process, how representative is GP in this problem?\n- Is it possible to use other surrogate models, like deep GP?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "18"}, "extra_text": {}}, {"review_id": "H3rA2EPy896", "reviewer": "Reviewer_Ct7A", "summary": "This paper addresses the Dynamic Causal Global Optimization (DCGO) problem that aims to find both the optimal set of manipulative variables and the optimal intervention levels to the selected variables. This is a time-series optimization that determines the best way to intervene in the present based on the results of interventions made in the past, and is considered to be a formulation of a particularly important problem e.g. in the medical field.\nThe authors make three assumptions about the DCGO problem:\n(1) the causal structure is invariant with respect to time, \n(2) the function value of a variable Y_t is determined from the function value of its parent node, and the function value of the parent node can be expressed as the sum of the function value of the target node and that of the non-target node, \n(3) the DAG does not contain any unobserved confounding factors. \n\nUnder these assumptions, it follows that the objective function of DCGO at time t can be expressed recursively using the results of interventions up to time t-1, and that the search space of DCGO is invariant with respect to time.\nUnder the above problem setup, the proposed method DCBO is an EI-based Bayesian optimization algorithm based on the Dynamic Causal Gaussian Process model whose mean and covariance function are defined by the recursive representation of the objective function.  \n\nExperiments to validate the effectiveness of the proposed method have been conducted on both synthetic and real data. Experiments with synthetic data have been conducted in six situations: when DAGs and SCMs are stationary/non-stationary with respect to time, when the manipulative variables contain noise, when there are missing observations, when there is no causal relationship among manipulative variables, \nand when there are multiple intervening variables.\nTwo real data experiments were conducted: one on the problem of minimizing the unemployment rate in a closed economy (economics), and the other on the problem of finding an intervention that would reduce the phenomenon of a concentration of dead animals in a chemostat (biology).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper aims at the optimization of sequential interventions, i.e., identifying the variables to be intervened and determining the optimal way to intervene. \nThis involves important problems, such as stage-by-stage optimization of multi-stage treatments in medicine. I think it is an important work to formulate this as an optimization problem under uncertainty, and to propose a framework for automatically searching for optimal interventions using a Bayesian optimization-type algorithm.\n\nQuality: This paper technically sounds and the performance of the proposed method has been verified in various situations including real data. In particular, settings where there is no causal relationship between the manipurative variables that would disadvantage the proposed method are also considered (in fact, the existing methods outperforms the proposed method in terms of accuracy), and thus it seems that fair comparisons are conducted. Theoretically, two properties Theorem 1, which states that the objective function can be written recursively, and Proposition 3.1, which states that the search space is time-invariant, may be important in Bayesian optimization, a problem that involves optimizing a function that is expensive to begin with.\n\nClarity: This paper is well organized. It clearly explains problem formulation, why a general framework for dynamic causal global optimization is needed, the process of deriving the proposed method, and its theoretical considerations.\n\nSignificance: As mentioned above, the problem considered in this study may have important applications. For example, in terms of medical applications, a reinforcement learning-based method called \"dynamic treatment regime\" is being considered as another approach to sequential optimization of interventions. However, this method has several problems that need to be solved before it can be put to practical use, such as being complicated of the method itself and the difficulty of estimation. There are still very few examples of its application to real problems. \nIt is expected that this and subsequent studies will provide a practical approach to these problems.\n\nQuestions \n\n- How realistic are the three assumptions (especially the first one that the causal structure is invariant with respect to time) introduced in section 2? Also, is it correct to interpret that the non-stationary case considered in the experiment deals with a situation that violates this assumption? In that case, I think the theoretical properties such as recursive representation of the objective function and constant search space are not guaranteed. How can we treat this?\n\n- How do you determine the initial state \\mathbb{M}_0　of the search space?\n\n- In the equation that define the GP prior, the notation \"m_{s, t}=... \\wedge k_{s, t}=...\" is confusing and better to be changed. \ne.g. m_{s, t}=..., k_{s, t}=...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors also consider experiments in problem settings where the proposed method does not work well.\nI believe that there is no negative social impact from this research.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20h"}, "extra_text": {}}, {"review_id": "AqRu1iLJarV", "reviewer": "Reviewer_YSDW", "summary": "The paper proposes a model, Dynamic Causal Bayesian Optimization (DCBO), which builds upon existing models to find a sequence of interventions , optimizing the target variable at each time step in a causal system. It combines Causal Bayesian Optimization, which builds around casual information and finds the optimal intervention in a DAG without temporal evolution, with Dynamic Bayesian Network, which is used in time-series modelling and carries dependence assumptions that don’t imply causation. DCBO represents associations between features and the causality between input and outputs, both of which can change over time. The modeling therefore offers a novel approach for decision making in dynamic systems ie. find optimal intervention at time t. The paper compares the model’s results against algorithms which either account for temporal evolution or optimize the sequence of interventions of a system (as no model exists which does both). The algorithm is run on both interventional and observational data. Finally, synthetic experiments are performed, such as creating noisy variables, incorporating missing observational data, and creating non-stationary settings. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes the modeling with mathematical rigor and precision. However, given the assumptions with which the framework is built, it is doubtful whether the approach could be used in any real-world scenario. Specifically, the modeling requires that the optimizing agent is allowed to perform a number of explorative interventions which do not change the underlying state of the system. It is difficult to identify a use case that satisfies this requirement. In section 4.2 (Line 342-351), the author provides “real experiments” that empoy real data to train a simulator/a system of ODE, which is then optimized. This is not an example of a “real experiment”.\nThe submission also does not elucidate how the proposed model identifies optimal interventions or treatment over a given time horizon.\nThe theoretical results obtained for both the synthetics and the real dataset do not show a substantial improvement from the already existing casual optimization models, with only slight improvements.\nOn a minor point, the mathematical notations are a bit cumbersome, and it is easy to get lost in a great variety of subscripts and quotes. Including a table summarizing the notation in one place would be particularly helpful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One of the paper's main points is to show the advantage of introducing a dynamic extension to existing causal Bayesian optimization methods. However, in line 145, invariance of the causal structure is assumed. This is a really restrictive hypothesis that severely hinders the possible application of this framework.  \nTo be fair, the submission states that some of the issues we point out will be addressed in future work. However, this framework could not be applied to real-world problems without such work. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "-HZu5jcDr6l", "reviewer": "Reviewer_o1ra", "summary": "In this paper, the authors extend the notion of casual Bayesian optimisation from a static to a dynamic scenario which, indeed, has many relevant applications. The paper theoretically analyses the proposed approach and run the algorithm in a set of real-world and synthetic data. This work is really interesting but I have some questions related to scalability and applicability that I hope the authors could help me understand. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1) I found definition 1 a bit confusing in some places: \n1.1.) When the set of observed variables V are defined, the authors write that those variables in the model are determined by variables in U \\cup V. So, the variables in V are defined by variables in the union of U and V. I am a bit confused about this as this somehow defies the set V through the set V.\n1.2.) When the set of functions f_i is introduced, the authors wrote that each of those is mapping from the union U_i and Pa(U_i) to V_i, where Pa(U_i) is a subset of V. Hence, from this definition, it follows that mapping f_i for each element in the union U_i and Pa(U_i) should put in correspondence a value of variable V_i. However,  f_i(Pa(u_i), u_i) is then defined as a function for an ordered pair of elements Pa(u_i) and u_i. Wouldn't this mean that the function f_i is defined on the cartesian product Pa(U_i) x U_i rather than the union P(U_i) \\cup U_i?\n1.3.) When the authors define the entire set F of functions f_1,.... f_n it is said that F forms a mapping from U to V. But each f_i is mapping not just from U but also it takes as an input element from Pa(U)\\in V. Hence, should not F be a mapping from U x V to V? \n\n2) Similar to standard assumptions in causal BO, the authors assume a known graph G. It is already rather hard to satisfy this assumption is standard causal BO without explicit prior knowledge. In this paper, does not this assumption entail even more prior knowledge? Shouldn't we also know how the graphs are varying over time as well?\n\n3) The algorithm proposed is interesting. I am wondering if the authors might reframe this as an instance of reinforcement learning and solve the corresponding problem? In other words, can there exist any relations to reinforcement learning and/or dynamic programming? If so, how can I think of the policy? Would the current algorithm correspond to a greedy policy for example? \n\n4) In formulation (1) for DCGO, if we are allowed to execute interventions on all variables X_t, will it not be the best option for us to optimise Y_t? The reason I ask is that although the cost of intervention is introduced it is not a involved in the objective function (but involved in the acquisition function)?\n\n4) Assumption 2, how restrictive is the additive assumption? In the experiments do we consider problems with such function f_{Y_t} and if yes, do we have a mechanism to check this property somehow?\n\n5) How the authors restrict the search space was not clear to me. My confusion stems from assumption 1, wherein the dynamic case the search space of subsets is the same as in the static case. Still, it is not clear, what is the size of this collection in this static case? Is it the cardinality of the power set, i.e. 2^{|X_t|}?\n\n6) Maybe removing the and operation from line 217 can aid readability. \n\n7) Can the authors elaborate/study the effect of the topology of the underlying DAG graph on the overall performance of the algorithm? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the paper can benefit from a larger study of social impacts especially after motivating health-care in the abstract.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an extension of causal Bayesian optimization from static to dynamic scenario to find a sequence of optimal interventions in a dynamic causal system. The problem formulation and the method are novel. In addition,  the results are well supported by experiments.  In general, the paper is well written. All of reviewers agree that the paper contains interesting contributions that are deserved to be presented in the conference. 


Decision: Accept (Poster)