BayesIMP: Uncertainty Quantification for Causal Data Fusion
Authors: Siu Lun Chau
Jean-Francois Ton
Javier Gonzalez
Yee Whye Teh
Dino Sejdinovic
Venue: NeurIPS 2021 Poster (2021)
Keywords: Kernel Methods
Causal Inference
Uncertainty Quantification

Abstract: While causal models are becoming one of the mainstays of machine learning, the problem of uncertainty quantification in causal inference remains challenging. In this paper, we study the causal data fusion problem, where data arising from multiple causal graphs are combined to estimate the average treatment effect of a target variable. As data arises from multiple sources and can vary in quality and sample size, principled uncertainty quantification becomes essential. To that end, we introduce \emph{Bayesian Causal Mean Processes}, the framework which combines ideas from probabilistic integration and kernel mean embeddings to represent interventional distributions in the reproducing kernel Hilbert space, while taking into account the uncertainty within each causal graph. To demonstrate the informativeness of our uncertainty estimation, we apply our method to the Causal Bayesian Optimisation task and show improvements over state-of-the-art methods.

BAYESIMP: Uncertainty Quantification for Causal Data Fusion
Siu Lun Chau University Oxford
Jean-François Ton' University of Oxford
Javier González
Microsoft Research Cambridge
Yee Whye Teh University of Oxford
Dino Sejdinovic
University of Oxford
Abstract
While causal models are becoming of the mainstays of machine learning, the problem of uncertainty quantification in causal inference remains challenging. In this paper, we study the causal data fusion problem, where datasets pertaining multiple causal graphs are combined to estimate the average treatment effect of target variable. As data arises from multiple sources and can vary in quality and quantity, principled uncertainty quantification becomes essential. To that end, introduce Bayesian Interventional Mean Processes, a framework which combines ideas from probabilistic integration and kernel mean embeddings to represent interventional distributions the reproducing kernel Hilbert space, while taking into account the uncertainty within each causal graph. To demonstrate the utility of our uncertainty estimation, we apply our method the Causal Bayesian Optimisation task and show improvements over fate-of-the-art methods.
Introduction
Causal inference has seen significant surge of research interest in areas such as healthcare [1], ecology [2]. and optimisation [3]. However, data fusion, the problem of merging information from multiple data sources, has received limited attention in the context of causal modelling, yet presents significant potential benefits for practical situations [4, 5J. In this work, we consider a causal data fusion problem where two causal graphs are combined for the purposes of inference of target variable (see Fig. 1). In particular, our goal is to quantify the uncertainty under such a setup and determine the level of confidence in our treatment effect estimates.
BMI
Cancer
Aspirin
Gleason
Target Variable
Cancer Volume
Mediating Variable
Treatment Variable
Age
Statin
PSA
PSA
Confounding Variable
Figure 1: Example problem setup: Causal graphs collected in two separate medical studies i.e. [6] and [7]. (Left) D, Data describing the causal relationships between statin level and Prostate Specific Antigen (PSA). (Right) D2 Data from a prostate cancer study for patients about to receive a radical prostatectomy. Goal: Model E[Cancrr Volume do(Statin)) while also quantifying its uncertainty.
Let us consider the motivating example in Fig. where a medical practitioner investigating how prostate cancer volume is affected by statin drug dosage. We consider the case where the doctor
35th Conference Neural Information Processing Systems (NeurIPS 2021).
"Denotes equal contribution with alphabetical ordering

---

only has access to two separate medical studies describing the quantities of interest. On one hand we have observational data, from one medical study D [1], describing the causal relationship between statin level and prostate specific antigen (PSA), and on the other hand we have observational data, from a second study D2 [7], that looked into the link between PSA level and prostate cancer volume. The goal is to model the interventional effect between our target variable (cancer volume) and the treatment variable (statin). This problem setting is different from the standard observational scenario as it comes with the following challenges:
Unmatched data:
Our goal is to estimate Elcancer volume do (statin)) but the observed cancer volume is not paired with statin dosage. Instead, they are related via mediating variable PSA.
Uncertainty quantification: The two studies may be of different data quantity/quality. Furthermore, a covariate shift in the mediating variable, i.c. a difference between its distributions in two datasets, may cause inaccurate extrapolation. Hence, we need to account for uncertainty in both datasets.
Formally, let X be the treatment (Statin). Y be the mediating variable (PSA) and T our target (cancer volume), and our aim is to estimate [[ (X)). The problem of unmatched data in a similar context has been previously considered by [5] using two-staged regression approach (X - Y and Y - T). However, uncertainty quantification, despite being essential our estimates of interventional effects will guide decision-making has not been previously explored. In particular, it is crucial to quantify the uncertainty in both stages as this takes into account the lack of data in specific parts of the space. Given that we are using different datasets for each stage, there also two sources of epistemic uncertainties (due to lack of data) as well as two sources of alcatoric uncertainties (due to inherent randomness in Y and T) [8]. is thus natural to consider regression models based on Gaussian Processes (GP) [9], as they are able to model both types of uncertainties. However, as GPs, or any other standard regression models, are designed to model conditional expectations only and will fail to capture the underlying distributions of interest (e.g. if there is multimodality in Y as discussed in [10]). This is undesirable since, as we will see, interventional effect estimation requires accurate estimates of distributions. While one could in principle resort to density estimation methods, this becomes challenging since we typically deal with a number of conditional/ interventional densities.
In this paper, we introduce the framework of Bayesian Interventional Mean Processes (BAYESIMP) to circumvent the challenges in the causal data fusion setting described above. BAYESIMP considers kernel mean embeddings [11] for representing distributions in a reproducing kernel Hilbert space (RKHS), in which the whole arsenal of kernel methods be extended to probabilistic inference (e.g. kernel Bayes rule [12], hypothesis testing [13), distribution regression [14]). Specifically, BAYES- IMP uses kernel mean embeddings to represent the interventional distributions and to analytically marginalise out Y, hence accounting for aleatoric uncertainties. Further, BAYESIMP uses GPs to estimate the required kernel mean embeddings from data in Bayesian manner, which allows to quantify the epistemic uncertainties when representing the interventional distributions. To illustrate the quality of our uncertainty estimates, we apply BAYESIMP to Causal Bayesian Optimisation [15]. an efficient hcuristic to optimise objective functions of the form x = arg m do()) = x). Our contributions are summarised below:
1. We propose novel Bayesian Learning of Conditional Mean Embedding (BAYESCME) that allows us to estimate conditional mean embeddings in a Bayesian framework.
2. Using BAYESCME, we propose a novel Bayesian Interventional Mean Process (BAYESIMP) that allows to model interventional effect across causal graphs without explicit density estimation, while obtaining uncertainty estimates for [[T]do(X) = z].
3. We apply BAYESIMP to Causal Bayesian Optimisation, problem introduced in [15] and show significant improvements over existing state-of-the-ard methods.
Note that [16] also considered a causal fusion problem but with a different objective. They focused on extrapolating experimental findings across treatment domains, i.e. inferring E[Y]do( when only data from p(Y (o))) [ is observed, where S is some other treatment variable. In contrast, we focus on modelling combined causal graphs, with a strong emphasis on uncertainty quantification. While [17] considered mapping interventional distributions in the RKHS to model quantities such as
they only considered a frequentist approach, which does not account for epistemic uncertainties.

---

Notations. We denote X,Y, Y, Z as random variables taking values in the non-empty sets X. 3 and Z respectively. Let
Z O
ky X' x X + R be positive definite kernels on X with an associated RKHS Hs.,- The corresponding canonical feature map (x', is denoted as (x²). Analogously for Y and Z.
x
YO OT
D.
NN (e, 2, 34
D2 = (./, 43
In the simplest setting, we observe i.i.d samples Dj (Ti, Mi, xiji=111 from joint distribution Pxyz which we con- catenate into vectors x ==
Figure 2: general two stage causal learning setup.
'N Similarly for y and z. For this work, X is referred as treatment variable, Y as mediating variable and Z as adjustment variables accounting for confound- ing effects. With an abuse of notation, features matrices are defined by stacking feature maps along the columns, i.e 2x := (x1) ..., [xN]]]] We denote the Gram matrix Kxx == 2x 4x and the vector of evaluations kax as (x, .................... kig (2r, xN)]. We define By. 42 analogously for and z.
our target variable, which is modelled as some noisy evaluation of a function f : 3 + T on Y while e being some random noise. For our problem setup we observe second dataset of i.idd realisations D2 {ūj. t 1 from the joint Pyr independent of D1. Again, we define == (ū,...., Das] and == [f,..., Las] just like for Dj. See Fig.2 for illustration.
Lastly, we denote T f(Y)
Background
Representing interventional distributions in RKHS' has been explored in different contexts [18, 17, 19]. In particular, when the treatment is continuous, [17] introduced the Interventional Mean Embeddings (IMEs) to model densities in RKHS by utilising smoothness across treatments. Given that IME is an important building block to our contribution, we give it a detailed review by first introducing the key concepts of do-caleulus [20] and conditional mean embeddings [21].
2.1 Interventional distribution and do-calculus
In this work, we consider the structural causal model [20] (SCM) framework, where a causal directed acyclic graph (DAG) G is given and encodes knowledge of existing causal mechanisms amongst the variables in terms of conditional independencies. Given random variables X and Y, a central question in interventional inference [20] is to estimate the distribution p(Y ((Y])o(X)
z), where (do(X) = x} represents an intervention on X whose value is set to x. Note that this quantity is not directly observed given that we usually only given observational data, i.e. data sampled from the conditional p XX but not from the interventional density p(Y (Y)do(XX ddo())) However, Pearl [20] developed do-calculus which allows observational distributions under the identifiability assumption. Here we present the backdoor and front-door adjustments, which are the fundamental components of DAG based causal inference.
estimate interventional distributions from purely
The backdoor adjustment is applicable when there are observed confound- ing variables Z between the cause X and the effect Y (see Fig. 3 (Top)). In order to correct for this confounding bias we can use the following equa- tion, adjusting for Z as p(Y [do(X) I) J'z p X x, 2))(2)ddz
Z
Z
X O
Y
The front-door adjustment applies to cases when confounders are un- observed (see Fig. 3 (Bottom)). Given a set of front-door adjustment variables Z, we can again correct the estimate for the causal effect from X to Y with p(Y |do X) I) Se Jxx p(Y|2² |2', 2)p(2|XX
(>)'''' dz.
We rewrite the above formulae in a more general form as show below. For the remainder of the paper we will opt for this notation:
Figure 3: (Top) Backdoor adjustment (Bottom) Front-door adjustment, dashed edges denote unobserved confounders.
pY X) = =x) F2, (V.100)))
For backdoor we have S.,, = (X = xr 2) and p(22.) =8pp(Z where 5, is the Dirac measure at X = x. For front-door, Ω2x = (X', 2} and p(f.x) = p (XX))(ZXX = x))
We refer the reader detailed review of RKHS methods provided in the Appendix of [5]
3

---

2.2 Conditional Mean Embeddings
Kernel mean embeddings of distributions provide a powerful framework for representing probability distributions 21] in an RKHS. In particular, we work with conditional mean embeddings (CMEs) in this paper. Given random variables X, Y with joint distribution P FYY the conditional mean embedding with respect to the conditional density p(||X- = x), is defined as:
/ 0n (y - x)dy
X= == Eyix==100
(2)
CMEs allow us to represent the distribution p(|| x)) as an element μy|x=z the RKHS Hay without having to model the densities explicitly. Following [21], CMES can be associated with Hilbert-Schmidt operator Cy,x: : Hrg + Hx,, known as the conditional mean embedding operator,
which satisfies _Y|x= = (x) where Cylx == CyxCxx with Cyx == x (Y) @ ox(X)) and Cxx == x0x @ Øx (X)] being the covariance operators. As a result, the finite sample estimator of Cyxx based on the dataset fx. can be written as:
Cy = 4y + A1) 4
(3)
where À > 0 is regularization parameter. Note that from Eq.3, [22] showed that the CME can be interpret as vector vector-valued kernel ridge regressor (V-KRR) i.c. o, (x) is regressed to an element in Hs.,- This is crucial as CMEs allow us to turn the integration, hence remove the need for explicit density estimation. This insight is important as it allows to derive analytic forms for our algorithms. Furthermore, the regression formalism of CMEs motivated us to derive Bayesian version of CME using vector-valued Gaussian Processes (V-GP), see Sec.3.
Eq.2. into regression task and
2.3 Interventional Mean Embeddings
Interventional Mean Embeddings (IME) [17] combine the above ideas to represent interventional distributions in RKHSs. We derive the front-door adjustment embedding here but the backdoor adjustment follows analogously. Denote (yo(d()=== as the IME corresponding to the interventional distribution p(Y)do((())))) [do(X) æ), which can be written as:
HY|do(X)== = / oy (v)p(ulld)( = x)dy fr Js (oov)pllle. p(z p(2(z)p(()) (())x))(a=))dddd Jdzdz'
CME jay X=a,Z=s
using the front-door formula with adjustment variable Z, and rearranging the integrals. By definition of CME f (v)(()(()) 2)dy = Crix,z (0x (x') @ dz (2)) and linearity of integration, we have
= =Crxx
the (h (aa))(('')ddl da' 09 Φz (=())((2())d Crxx,z 8 8
Using notations from Sec.2.1, embedding interventional distributions into an RKHS is follows.
Proposition 1. Given an identifiable do-density of the form p(Y dd( =) Ea, [p(([ the general of the empirical interventional mean embedding is given by.
(y)do(()==) = (K + (x)
(4)
where K2.- = -xxx 3 Kzz and D2 (x)) is derived depending on p(ltg). ). In particular, for backdoor adjustments. 1 (bd) (x) 1 k (z, -) O 1 fs and for front-door 4 (fd)
S2 (x) 4 x O 2 12
3 Our Proposed Method
Two-staged Causal Learning. Given two independent datasets D, = [(2,, Zi, 9:) and D2 = (Gi, 4 our goal to model the average treatment effect when intervening on variable X, ie model g(x) = do)))o(X) = ær). Note that the target variable T and the treatment variable X are never jointly observed. Rather, they are linked via mediating variable Y observed in both datasets.

---

Model
/ as KRR
METHODS
Stage
Stage 2
IME [17]
KRR
KRR
X
Hh.
IMP (Ours)
KRR
GP
BAYESIME (Ours)
GP
KRR
Bryadan
Par
GP
BAYESIMP (Ours)
GP
GP
Make
Figure 4: Two-staged causal learning problem
Table 1: Summary of our proposed methods
In problem setting, make the following two assumptions: (A1) The treatment only affects the target through the mediating variable, i.c T IL do(X) + PoT)do(X) Y) = p(T) Y). in other words, that in the true data generating model P(X, Y, 7. T), all causal paths from X to T are mediated through Y. and (A2) Function f given by fVy = FE[TYY = y| belongs to an RKHS He.
We can thus express the average treatment effect as:
g(x) = (o)) = z] = / E = x, - = = z)dy
(5)
JTTY w/. T1 do(
| f f()pp)ldd(x)) = =xddy----- = (f. HY
=
(6)
The final expression decomposes the problem of estimating y into that of estimating the IME jy do(X) (which can be done using D1) and that of estimating the integrand f -> 'T (which can be done using D2). Each of these two components can either be estimated using a GP KRR approach (See Table 1). Furthermore, the reformulation as an RKHS inner product is crucial, as it circumvents the need for density estimation as well the need for subsequent integration in Eq.6 Rather, the main parts of the task can now be viewed as two instances of regression (recall that mean embeddings can be viewed as vector-valuee regression).
To model and quantify its uncertainty. we propose 3 GP-based approaches. While the first 2 methods, Interventional Mean Process (IMP) and Bayesian Interventional Mean Embedding (BAYESIME) are novel derivations that allow us to quantify uncertainty from either one of the datasets, we treat them as intermediate yet necessary steps derive main algorithm, Bayesian Interventional Mean Process (BAYRSIMP), which allows us to quantify uncertainty from both sources in a principled way. For summary of the methods, see F and Table All derivations are included in the appendix.
Interventional Mean Process: Firstly, we train as a GP using D2 and model do))d(X)== as V- KRR using D1. By drawing parallels Bayesian quadrature [24] and conditional mean process introduced in [25], the integral of interest g(x) = f(y)p(v do(X) = x dy will be a GP indexed by the treatment variable X. We can then use the empirical embedding ( learnt in Dj to obtain an analytic mean and covariance of g.
Bayesian Interventional Mean Embedding: Next, to account for the uncertainty from D1, we model as a KRR and =))= using V-GP. We introduce our novel Bayesian Learning of Conditional Mean Embeddings (B (BAYESCME), which uses a nuclear dominant kernel [26] construction, similar to [27], to ensure that the inner product (f, (d)))x()==?? is well-defined. As the embedding a GP, the resulting inner product is also GP and hence takes into account the uncertainty in DSee Prop. 4).
Bayesian Interventional Mean Process: Lastly, in order to account for uncertainties coming from both D2 and D2, we combine ideas from the above IMP and BAYESIME. We place GPs on both and do(X) and use their inner product to model E[]]d Interestingly. the resulting uncertainty can be interpreted as the sum of uncertainties coming from IMP and BAYESIME with an additional interaction term (See Prop.5).
3.1 Interventional Mean Process
Firstly, we consider the case where f is modelled using a GP and JoY [do[X)== using a V-KRR. This allows us take into account the uncertainty from D:, by modelling the relationship between and
We note that this two-staged setup resembles Instrumental Variable [23] (IV) regression. However, our general setup allows the IV and the treatment to be confounded, which is not the case in standard IV regression.
5

---

T using in a GP. Drawing parallels to Bayesian quadrature [24] where integrating f with respect to a marginal measure results into a Gaussian random variable. we integrate f with respect a conditional measure, thus resulting in a GP indexed by the conditioning variable. Note that [25] studied this GP non-causal setting, for a very specific downscaling problem. In this work, we extend their approach to model uncertainty in the causal setting. The resulting mean and covariance are then estimated analytically, ie without integrals, using the empirical IME (i]ddo(X) learnt from Di, see Prop.2.
Proposition 2 (IMP). Given dataset D1 = {1 Vi, 21) and D2 = ((0,, if f is the posterior GP learnt from D2, then g = J (u)ppyl Applyyl(Xyy dyy is GP GP(m1, K1) defined on the treaiment variable X with the following mean and covariance estimated using ())
mi m1(x) - (KY(do((s) my) Ln, (x) (Kn, + +yy-KKyy (Kyg + +
(7)
K1 (xx,xx)) ) flyldo(x') -yl/dole) (y + >1)- (d(()(())) -Dn (x) (1)
(8)
T (Ka, + +11)-1 +Xl)-IKyyll (Kn, + 11)
(9)
where (o))d(()) (á(()))) Kyy = D. my and Kyy are the posterior mean function and covariance of f evaluated at respectively. x 0 is the regularisation of the CME. Xj > 0 is the noise term for GP f. Sa is the set of variables as specified in Prop.
The posterior covariance between x and x' in IMP can be interpreted as the similarity between their corresponding empirical IMES [y]do(()== and (oyjdo weighted by the posterior covariance Kyy, where the latter corresponds to the uncertainty when modelling f as GP in D2- However, since f only considers uncertainty in D2, we need to develop a method that allows us to quantify uncertainty when learning the IME from Dj. In the next section, introduce a Bayesian version of CME, which then lead to BAYESIME, a remedy to this problem.
Summary:
3.2 Bayesian Interventional Mean Embedding
To account for the uncertainty in Dj when estimating d)))))) we consider GP model for CME, and later extend to the interventional embedding IME. We note that Bayesian formulation of CMEs has also been considered in [28]. but with specific focus on discrete target spaces.
Bayesian learning of conditional mean embeddings with V-GP. As mentioned in Sec.2, CMEs have clear "feature-to-featie By (Y) onto (X). Hence, we consider a valued GP construction to estimate the CMM.E
regression perspective, i.e (Y) = 2 is the result of regressing
Let Pap(x, y) be a GP that models (v)- Given that J E Has, for (S Pop(a, -) Ha, to be well defined, we need to ensure (2, :) is also restricted to Hs, for any fixed x. Consequently, we cannot define a GP(O, kn @ kyy) prior on Agp as usual, as draws from such prior will almost surely fall outside He. He, [26]. Instead we define prior over M'p ~ GP k2 ry), where ry is a nuclear dominant kernel [26] over ky. which ensures that samples paths of ('gp live in Hi, & Hy, almost surely. In particular, we follow a similar construction as in [27] and model ry as ry (yte, 31y) ʃ kig (yi, u)kk( (u, yj (d()) where is some finite measure on Y. Hence we can now setup a vector-valued regression in He, as follows:
(1,)) = Hap (2,: .) AåG
(10)
where ~ r) are independent noise functions. By taking the inner product with (3/) on
both sides, we then obtain (y,, v²) Ma (R2, y²) + c,'). Hence, we can treat k(y,, U2) as noisy evaluations of Mgp Mj) and obtain the following posterior mean and covariance for Mgp-
Proposition 3 (BAYESCME). The posterior GP of Ugg given observations {x, y) has the following mean and covariance:
mp (p, v)) = kix (Kxx + AT) Kyy
(11)
F y): (1)) = T - kax ( (xxx + - Kx Ryg' yo'
(12)
In addition, the following marginal likelihood can be used for hyperparameter optimisation,
N 2
log Kax + AM| log (RI) Tr ((Kex + 21) "or'Kyylll Kyy Kyv)
1
(13)
6

---

Note that in practice we fix the lengthscale of ky and ry when optimising the above likelihood. This is avoid trivial solutions for the vector-valued regression problem discussed in [10]. The Bayesian version of the IME is derived analogously and we refer the reader to appendix due to limited space.
Finally, with V-GPS on embeddings defined, we can model g(x)) as (J, (ggll 1) Hr,, which due to the linearity of the inner product, is itself a GP. Here, we first considered the case where is KRR learnt from D2 and call the model BAYESIME.
Proposition 4 (BAYESIME). Given dataset D1 = {(23, Yis 2 and D2 = ((5.,t)))3 f a KRR leartt from D2 and (yjdo(X) modelled as a V-GP using Dj. then g = (f, Hy]do(x)) ~ GP(m2, K2) where,
Rys 62(2, x') 14 (x) Dn (x) CDn (1) (Knr A1)
m2(x)= = Ln, (x)
+
(14)
- 1 (x))
(15)
where A = [K3s + X1 = A Ryg and C = A Roy Royy Ryy Rys
Summary: Constants B and C' in K2 can be interpreted as different estimation of [lf[ i. the RKHS norm of f. As result, for problems that are "harder" to learn in D2, i.e. corresponding to larger magnitude of li,,, will result into larger values of B and C. Therefore the covariance Kig can be interpreted uncertainty in Dj scaled by the difficulty of the problem learn in D2.
3.3 Bayesian Interventional Mean Process
To incorporate both uncertainties in DD and D2, we combine ideas from IMP and BAYESIME to estimate g (J, ("y]do(X) by placing GPs on both and [dy]do(X)- Again before, a nuclear dominant kernel ry was used to ensure the GP f is supported on H2,- For ease of computation, we consider a finite dimensional approximation of the GPs f and jy [do(X) and estimate g as the RKHS inner product between them. In the following we collate y and y into single set of points y, which can be seen as landmark points for the finite approximation [29]. We justify this in the Appendix.
Proposition 5 (BAYESIMP). Let f and py be GPs learnt as above. Denote ff and (oy))(()) as the finite dimensional approximation of f and do(x) respectively. Then (f do())) has the following mean and covariance:
m²(x)= = Fyy Ky K-1 Rys (Rys + +t yÿ
(16)
(a(xx2²) = Eg OT Rys O + + (00 Pex' - 0 Go Gra + +0 Fix - - Gez! Gez!
(17)
Uncertainty from Dr
Uncertainty from D2
Uncertainty from Interaction
where Es Dn (x) (Kn, + A1)-1 Frw Dn. (x) Dn (x'), Gaz Q2 (x) (Kn.
(x'), and 0₁ = -Kgg Ryy Kyy, ((4)) O]] Ryg 01 = O Rey Ryy Ryy O1
1 and O = tr( tr(Kz) gy Ryy Kyg 1
sy Ryy)) 9²¹² = tr(Rgy Ryy Rys 38 Ros Kzz)) and Os =
Kzg Rys (Kyg + A1) Rgg is the posterior covariance of evaluated at ý
Summary: While the first two terms in R3 resemble the uncertainty estimates from IMP and BAYESIME, the last term acts as an extra interaction between the two uncertainties from Dj and D2. We note that unlike IMP and BAYESIME, ğ from Prop.5 is not GP inner products between Gaussian vectors are not Gaussian. Nonetheless, the mean and covariance can be estimated.
4 Experiments
In this section, we first present an ablation studies on how our methods would perform under settings where we have missing data parts at different regions of the two datasets. We then demonstrate BAYESIMP's proficiency in the Causal Bayesian Optimisation setting.
In particular, we compare our methods against the sampling approach considered in [15]. [15] start by modelling f + T as GP and estimate the density p(Y [do(X)) using GP along with do-calculus. Then given treatment z, obtain L samples of yı and R samples of fr. from their posterior GPs.
7

---

(a) Sampling
(b) IMB
(c) IMP
(d) BayesIME*
(c) BayesIMP
ElTdooo -x0 x]
Extopiiatoo
Osabesared saice
Groundesh
0
5
Figure Ablation studies of various methods in estimating uncertainties for an illustrative experiment. indicates our methods. N = M = 100 data points are used. Uncertainty from sampling gives a uniform estimate of uncertainty and IME does not come with uncertainty estimates. We see IMP and BAYESIME covering different regions of uncertainty while BAYESIMP takes the best of both worlds.
The empirical mean and standard deviation of the samples (vr 1=1.r=1 { can now be taken to estimate (X) = x] as well as the correspondingly uncertainty. We emphasize that this point estimation requires repeated sampling and thus inefficient compared to our approaches, where we explicitly model the uncertainty as covariance function.
Ablation study. In order to get better intuition into our methods, we will start off with preliminary example, where we investigate the uncertainty estimates in a toy case. We assume two simple causal graphs X Y for D1 and Y + T for D:2 and the goal is to estimate EET (dX)) = x (generating process given in the appendix). We compare our methods from Sec.3 with the sampling-bai -aseeg uncertainty estimation approach described above. In Fig.5 we plot the mean and the 95% credible interval of the resulting GP models for [[T]do(X) = z²|. On the z-axis we also plotted a histogram of the treatment variable r to illustrate its density.
From Fig.5(a),, we see that the uncertainty for sampling is rather uniform across the ranges of x despite the fact we have more data around x O. This is contrary to our methods, which show a reduction of uncertainty at high x density regions. In particular, x = -5 corresponds to an extrapolation of data, where x gets mapped to a region of y where there is no data in D2. This fact is nicely captured by the spike of credible interval in Fig.5(c) since IMP utilises uncertainty from D2 directly. Nonetheless, IMP failed to capture the uncertainty stemming from D1, as seen from the fact that the credible interval did not increase as we have less data in the region |z| > 5. In contrast, BAYESIME ((d) gives higher uncertainty around low x density regions but failed to capture the extrapolation phenomenon. Finally. BAYES Fig.5(e) seems to inherit the desirable characteristics from both IMP and BAYESIMÉ, due to taking into account uncertainties from both Di, D2. Hence, in the our experiments, we focus on BAYESIMP and refer the reader to the appendix for the remaining methods.
IMYesIMP for Bayesian Optimisation (BO). We now demonstrate, on both synthetic and real-world data, the usefulness of the uncertainty estimates obtained using our methods in BO tasks. Our goal is to utilise the uncertainty estimates to direct the search for the optimal value of [[T]do([X by querying as few values of the treatment variable X as possible, i.c. we want to optimize for xx = arg min the following two datasets: D1 = {æ,, Ug 24. V/i N and D: = {ūz, t f= Note that BAYESIMP from Prop.5 is not GP as inner products between Gaussian vectors are not Gaussian. Nonetheless, with the mean and covariance estimated, we will use moment matching to construct GP out of BAYESIMP for posterior inference. At the start, we are given D, and D2, where these observations are used to construct a GP prior for the interventional effect of on T, i.e E(T]do(X) = x). to provide a "warm" start for the BO.
z]
= xr. For the first synthetic experiment (see Fig.6 (Top)), we will use
Again we compare BAYESIMP with the sampling-bast based estimation of [E[T" [E[]) [X] and its uncertainty, which is exactly the formulation used in the Causal Bayesian Optimisation algorithm (CBO) [15]. In order to demonstrate how BAYESIMP performs in the multimodal setting, we will be considering the case where we have the following distribution on Y i.e. p(y) u. 2) TP1 (3/ u, 2) + (1 π T)p2 (y) u, 2) where is mixture and π E [o, 1]. These scenarios might arise when there is an unobserved binary variable which induces switch- ing between two regimes on how Y depends on (U, 2). In this case, the GP model of [15] would only capture the conditional expecta-
to
Figure 6: Illustration of syn- thetic data experiments.
8

---

tion of Y with an inflated variance, leading to slower convergence and higher variance in the estimates of the prior as we will show our experiments. Throughout our experiments, similarly to [15], we will be using the expected improvement (EI) acquisition function to select the next point to query.
Synthetic data experiments. We compare BAYESIMP to CBO as well as simple GP with no learnt prior as baseline. We will be using N = 100 datapoints for Dj and M = 50 datapoints D2. We ran each method 10 times and plot the resulting standard deviation for each iteration in the figures below. The data generation and details were added in the Appendix. We see from the Fig.7 7 that
Multinoodal (Frontdoor)
Multinsodll (Backdaoo)
Unimodal (Bockdoor)
Vante Vone-1o-qiim 10-7iiiiz
Ville Sovegg eepiniie
Valiec sovegeientecc
bazeline
CBO
Baye-IMP
(to
BO iuriioi
BD Nuraiion
BD Fecrationn
Figure 7: We are interested in finding the maximal value of [[T)do(X) x] with as few BO iterations as possible. We ran experiments with multimodality in Y. (Left) Using front-door adjustment (Middle) Using backdoor adjustment (Right) Using backdoor adjustment (unimodal Y)
BAYESIMP is able to find the maxima much faster and with smaller standard deviations, than the current state-of-the-an art method, CBO, using both front-door and backdoor adjustments (Fig.7(Right, Middle)). Given that our method uses more flexible representations conditional distributions, we able to circumvent the multimodality problem in Y. In addition, we also consider the unimodal version, i.e. π 0 (see right Fig.7). We see that the performance of CBO improves in the unimodal setting, however BAYESIMP still converges faster than CBO even in this scenario.
Next, we consider a harder causal graph (see Fig.6 (Bottom)), previously considered in [15]. We again introduce multimodality in the Y variable in order to explore the case of more challenging conditional densities. We see from Fig.8 (Left, Middle), that BAYESIMP again converges much faster to the true optima than CBO [15] and the standard GP prior baseline. We note that the fast convergence of BAYESIMP throughout our experiments is not due to simplicity of the underlying BO problems. Indeed, the BO with a standard GP prior requires significantly more iterations. It is rather the availability of the observational data, allowing us to construct a more appropriate prior, which leads "warm" start of the BO procedure.
Mullimedal (Backdoocc)
Multiredall (Frmndour)
Healiheee Data
Valte optim.
Value spptttniiz
hyotine
BayesEMIP
Groundrat
BO inniaii
BD iecation
Dentiim
Figure 8: (Left) Experiments where we are interested in [[T = d] with multimodal Y, (Middle) Experiments where we are interested in [[T]do(( E) e] with multimodal Y, (Right) Experiments on healtheare data where we are interested in BCCncer Volume do(Statin)).
Healthcare experiments. We conclude with a healthcare dataset corresponding to our motivating medical example in 1ig.1. The causal mechanism graph, also considered in the CBO paper [15], studies the effect of certain drugs (Aspirin/Statin) on Prostate-Speciffy Antigen (PSA) levels [6]. In our case, we modify statin to be continuous, in order to optimize for the correct drug dosage. However, in contrast to [15], we consider a second experimental dataset, arising from different medical study, which looks into the connection between PSA levels and cuncer volume amount in patients [7]. Similar to the original CBO paper [15], given that interventional data is hard to obtain, we construct data generators based on the true data collected [7]. This is done by firstly fitting a GP on the data and then sampling from the posterior (see Appendix for more details). Hence this is the

---

perfect testbed for our model where we are interested in E[Cancer Volume do(Statin)). We see from Fig.8 (Right) that BAYESIMP again converges to the true optima faster than CBO hence allowing us to find the fastest ways of optimizng cancer volume by requesting much less interventional data. This could be critical as interventional data real-life situations can very expensive to obtain.
Discussion and Conclusion
In this paper we propose BAYESIMP for quantifying uncertainty the setting of causal data fusion. In particular, our proposed method BAYESIMP allows us to represent interventional densities in the RKHS without explicit density estimation, while still accounting for epistemic and aleatoric uncertainties. We demonstrated the quality of the uncertainty estimates in a variety of Bayesian optimization experiments, in both synthetic and real-world healthcare datasets, and achieve significant improvement over current SOTA in terms of convergence speed. However, BAYESIMP is not designed to replace CBO but rather am alternative model for interventional effects.
emphasize that
In the future, we would like to improve BAYESIMP over several limitations. As in [15], we assumed full knowledge of the underlying causal graph, which might limiting in practice. Furthermore, as the current formulation of BAYESIMP only allows combination of two causal graphs, we hope to generalise the algorithm into arbitrary number of graphs in the future. Causal graphs with recurrent structure will be interesting direction to explore. Lastly, we would also like to include cost function as [15] to constrain the search space to the most sensible solutions.
Acknowledgements
The authors would like to thank Bobby He, Robert Hu, Kaspar Martens, Jake Fawkes and Joost van Amersfoort for helpful comments. SLC and JFT are supported by the EPSRC and MRC through the OxWaSP CDT programme EP/L0167 10/1. YWT and DS are supported in part by Tencent Al Lab and DS is supported in part by the Alan Turing Institute (EP/N500129/1 YWT's research leading to these results has received funding from the European Research Council under the European Union's Seventh Framework Programme (FP7/2007-2013) ERC grant agreement 617071.
References
[1] Clay Thompson. Causal graph analysis with the causalgraph procedure. In Proceedings of SAS Global Forum, 2019.
[2] Travis Courtney, Mario Lebrato, Nicholas Bates, Andrew Collins, Samantha De Putron, Rebecca Garley, Rod Johnson, Juan-Carlos Molinero, Timothy Noyes, Christopher L Sabine, al. Environmental controls on modern scleractinian coral and reef-scale calcification. Science advances, 3(1 :ell)ee1713336 2017.
[3] Virginia Aglietti, Theodoros Damoulas, Mauricio Álvarez, and Javier González. Multi-task causal learning with gaussian processes. Advances in Neural Information Processing Systems, 33, 2020.
[4] Tong Meng, Xuyang Jing, Zheng Yan, and Witold Pedrycz. A survey on machine learning for data fusion. Information Fusion, 57:1 11-119, 2020.
[5] Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. arXiv preprint arXiv: 1906.0023, 2019.
[6] Ana Ferro, Francisco Pina, Milton Severo, Pedro Dias, Francisco Botelho, and Nuno Lunet. Use of statins and serum levels prostate specific antigen. Acta Urológica Portuguesa, 32(2):71-77, 2015.
[7] Thomas A Stamey, John Kabalin, John McNeal, lain Johnstone, Fuad Freiha, Elise Redwine, and Norman Yang. Prostate specific antigen in the diagnosis and treatment of adenocareinoma of the prostate. ii radical prostatectomy treated patients. The Journal of urology, 14765):0066-108,,, 1989.
10

---

[8] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457-566, 2021.
[9] Rasmussen and C Williams. Gaussian Processes for Machine Learning. 2005.
[10] Jean-Francois Ton, Lucian Chan, Yee Whye Teh, and Dino Sejdinovic. Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 Proceedings of Machine Learning Research,
pages 1099-1107. PMLR, 13-15 Apr 2021.
[11] Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Berhaard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends@ in Machine Learning, 10(1 101,11):1-14, 2017.
[12] Kenji Fukumizu, Le Song, and Arthur Gretton. Kernel bayes' rule. arXiv preprint arXiv:1009.573k4 2010.
[13] Qinyi Zhang, Sarah Filippi, Arthur Gretton, and Dino Sejdinovic. Large-scale kernel methods for independence testing. Statisties and Computing, 28(1):113-130 2018.
[14] Ho Chung Leon Law, Dougal Sutherland, Dino Sejdinovic, and Seth Flaxman. Bayesian approaches distribution regression. In International Conference on Artificial Intelligence and Statistics, pages 167-1176. PMLR, 2018.
[15] Virginia Aglietti, Xiaoyu Lu, Andrei Paleyes, and Javier González. Causal bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 31 3155-3164. PMLR, 2020.
[16] Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy Sciences, 113(27):735--7352 2016.
[17] Rahul Singh, Liyuan Xu, and Arthur Gretton. Kernel methods for policy evaluation: Treatment effects, mediation analysis, and off- planning. arXiv preprini arXiv:2010.04855, 2020.
[18] Krikamol Muandet, Motonobu Kanagawa, Sorawit Saengkyongam, and Sanparith Marukatat. Counterfactual mean embeddings. arXiv preprint arXiv:/805.0884/ 2018.
[19] Jovana Mitrovic, Dino Sejdinovic, and Yee Whye Teh. Causal inference via kernel deviance measures. arXiv preprint arXiv:1804.0000 046622 2018.
[20] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-683 1995.
[21] Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine, 30(4):98-111, 2013.
[22] Steffen Grünewälder, Guy Lever, Luca Baldassarre, Sam Patterson, Arthur Gretton, and Mas- similano Pontil. Conditional mean embeddings as regressors-suppleeenary arXiv preprint arX5v: 2012.
[23] Krikamol Muandet, Arash Mchrjou, Si Kai Lee, and Anant Raj. Dual instrumental variable regression. arXiv preprint arXiv:1910.12355 2019.
[24] Trançois-Xavier Briol, Chris Oates, Mark Girolami, Michael A Osborne, Dino Sejdinovic, et al. Probabilistic integration: A role statistical computation? Statistical Science, 34(1):1-22, 2019.
[25] Siu Lun Chau, Shahine Bouabid, and Dino Sejdinovic. Deconditional downscaling with gaussian processes. arXiv preprint arXiv:2 2021.
[26] Milan Lukić and Jay Beder. Stochastic processes with sample paths in reproducing kernel hilbert spaces. Transactions of the American Mathematical Society, 353(10):3955-399, 2001.
11

---

[27] Seth Flaxman, Dino Sejdinovic, John P Cunningham, and Sarah Filippi. Bayesian learning of kernel embeddings. arXiv preprint arXiv:1633.02160, 2016.
[28] Kelvin Hsu, Richard Nock, and Fabio Ramos. Hyperparameter learning for conditional kernel mean embeddings with rademacher complexity bounds. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 227-242 Springer, 2018.
[29] Giancarlo Ferrari Trecate, Christopher KI Williams, and Manfred Opper. Finite-dimensional approximation of gaussian processes. In Proceedings of the 1998 conference on Advances in neural information processing systems II, pages 218-224, 1999.
12


Reviews:
[{"review_id": "duy3C_kYdDT", "reviewer": "Reviewer_wLHr", "summary": "The paper proposes a framework for embedding interventional distributions into RKHS that accounts for uncertainty in causal datasets especially when the confounders, treatments, mediators, and outcomes are not all observed in a single observational dataset. The main problem that is addressed here is quantifying the uncertainty in the estimation of causal effects when the treatment and the outcome are measured in two separate sources (potentially under different data-generating conditions) and the aim is to estimate the effect from a causal data fusion perspective. The paper introduces Bayesian Interventional Mean Processes for representing interventional distributions in the reproducing kernel Hilbert space. The method is applied to the task of Causal Bayesian Optimization where it is shown that the method converges faster as compared to other baseline methods. The main contribution of the paper is obtaining the uncertainty estimates for $E\\[T\\mid do(X) = x\\]$.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of quantifying uncertainty about causal estimates especially when the treatment and the outcome are not observed in the same setting is of significant importance but relatively understudied. The proposed approach definitely builds in this direction. Relation to prior work is discussed and cited when the approach builds on ideas developed therein.\n\nThe claims are well-supported to the best of my knowledge.  The write-up is generally clear. While the example on estimating the effect of statin drug dosage on cancer volume is helpful, it would also help the reader to present some of the real-world settings where the approach may not succeed as expected, for example, in the case of recommender systems with high-dimensional confounding and higher uncertainty about the causal graph as well.\n\nThe results will aid practitioners in quantifying the uncertainties with respect to the causal effect estimates as well as guide in better data-collection practices if the uncertainties span over a larger region and estimating the effects with data from two different sources may not be accurate. It would help to address/analyze with the setting where the data is from a single source but the causal graph is uncertain, this can be an interesting future direction as well. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Given that the proposed approach aims to combine causal information from multiple datasets that may not all be collected under the same settings, it can have several potential impacts in fields like healthcare. The paper discusses this to some extent.\n\nThe limitations are discussed with respect to assuming complete knowledge of the causal graph. However, there could also be challenges with obtaining enough samples across the two sources, in which case there would be additional uncertainty because of representation bias. It would be helpful to elucidate the specific challenges and extensions in this case.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "KnuiXJN2qqV", "reviewer": "Reviewer_7uEU", "summary": "The paper proposes a Bayesian approach for estimating the treatment effect by combining two data sources. The main advantage of the Bayesian approach over the frequentist counterpart [17] is its ability to quantify uncertainty. The main machinery was GP which replaces KRR used in existing works. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written, experiments are adequate to illustrate the strength of the approaches, and the proposed idea has some interesting contributions (i.e., causal uncertainty quantification in the data fusion context) to the existing literature. In terms of potential applications of the proposed methods, I believe there are plenty in medicine and other fields where data are collected from multiple sources.  Therefore, I believe overall the quality, clarity, and significance of this work are high.\n\nOne concern is, however, regarding the originality. The novel elements of this paper are all based on established techniques, e.g., the IMP extends [24] and IME extends [25-27]. The idea of using RKHS is also not particularly new. However, the utilization of these various existing techniques in a meaningful way in this particular context is, in my opinion, novel. \n\n======update=====\nThanks for the authors' responses. I keep my original score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are adequately discussed and the potential negative societal impact is briefly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "AWmz5uNEe3A", "reviewer": "Reviewer_ekhA", "summary": "This paper tackles the problem of computing the average causal effect given data generated by distinct (fragmented) graphs which does not contain the same set of variables. It uses kernel mean embeddings to represent interventional distributions, marginalize out the common variable Y and account for statistical uncertainties. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly the problem of fusing datasets consisting of different variables  from distinct data sources and using them to answer questions is a non-trivial  and important task. So this is an important and interesting work. \n\nMy main concern here is the assumption T || do(X)|Y. Note that T and Y are variables and do() is an operator. As such it is strange to see an independence claim expressed between  a variable and an operator. Normally, we rely on rules of do-calculus to remove or add do(x) operators to factors in an estimand. In this case the variables do not  belong to the same  causal model (but to two distinct graphs) and as such it is not clear how to directly apply do-calculus. \n\nA) By T || do(X)|Y, do you mean that in the true data generating model G that generated P(X,Y,Z,T) (which is unavailable), X is not a direct cause of T and that all causal paths from X to T are mediated by Y? If so, explicitly state this. If not, please explain this to me. \n\nB) Wouldn’t a minor modification of your results make them applicable to the case where Z is not a common cause of X and Y but related to them in the following way: X-->Z-->Y? \n\nC) Also in the model you have considered, why are there no confounders between Y and T? Realistically, wouldn't that be the case?\n\n     \nMore importantly, I think the authors need to put in a bit more effort to make their results more accessible to the general audience. Even in the case of causal inference researchers, not everyone can be expected to be familiar with RKHS and hence might not be able to make use of the results presented here. It is important therefore that the authors add a sentence or two after each proposition describing it in english (without RKHS jargon). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": " "}, "extra_text": {}}, {"review_id": "1SFZjSZGLNu", "reviewer": "Reviewer_JKck", "summary": "The paper studies the problem of uncertainty quantification for causal data fusion.  They introduce Bayesian Interventional Mean Processes, a framework that represents the interventional distributions using kernel mean embeddings and probabilistic integration. The framework is able to measure the uncertainty within the causal graph in each dataset. In experiments, the author demonstrates the performance improvement of the proposed method on the causal Bayesian optimization task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The setup is clearly related to the problem of instrumental variable (IV) regression. Besides the difference that the regression at different stages are based on different datasets, the connection with IV can be seen by changing the treatment variable to an instrumental variable and changing the mediating treatment variable to a treatment variable. \n\nIf this is the case, I think the method may be better suited to be presented in the potential outcome (PO) framework rather than the framework of structural causal models. Under the PO framework, you should provide the same assumptions for instrumental variable (IV) regression. For example, the treatment variable (the instrumental variable in IV) has no direct effect on the target variable; there are no unmeasured confounders for the treatment variable (the instrumental variable in IV) and the target variable. Because the paper is about uncertainty quantification, I found putting down these formal assumptions becomes more important. For instance, the biggest uncertainty in causal inference can be due to some unmeasured confounders of the treatment variable (the instrumental variable in IV) and the target variable. Later on, I see you provide two assumptions in lines 155-157. I do believe these assumptions should be made earlier, e.g., in Section 2.1.\n\nIn the introduction, I think the author should provide more detailed explanations of epistemic uncertainties and aleatoric uncertainties. Bayesian methods are not mainstream in causal inference. I don’t think causal inference researchers are familiar with these two concepts. In line 74, you mention “[17] considers a frequentist approach, which does not account for epistemic uncertainties”.  Not all the frequentist approaches measure the uncertainty using the asymptotic confidence intervals. For example, bootstraps or subsampling can be used to quantify uncertainties. It is unclear why we should use the Bayesian approach to quantify the uncertainty instead of the frequentist approach. It is possible that the user chooses a wrong prior on the Gaussian process models, then the Bayesian uncertainty is also not very well calibrated, i.e., the epistemic uncertainties are not quantified in the right class of models.\n\nThe proposed method is presented in a clear way. However, I don’t think making something Bayesian is very novel technically, given what has been done on Interventional mean embeddings and the connection between kernel ridge regression and Gaussian process regression. Is there any technical challenge due to using multiple datasets your method overcome but I didn’t realize?\n\nIn experiments, the ablation study should report the coverage rate at a range of sample sizes, dimensions and simulated functions. It is not convincing to see just the performance of one function, dimension and sample size. Causal Bayesian optimization is an interesting application but may be too complicated. I understand fast convergence in Bayesian optimization means the method estimates the function mean and variance very well. I expect to see some simple experiments on treatment effect estimation, in which you can disentangle the gain from estimating mean and variance. Also, Bayesian optimization is not effective at every iteration; it just improves the value to optimize in some steps. Overall, I think doing experiments on treatment effect estimation directly is more transparent. For your healthcare experiments, learning dosage in practice is usually not that straightforward. Increasing the dose level also increases the toxicity. There should be an unknown safety constraint on your intervention. Bayesian optimization in your healthcare example should take this into account for the method to be used in real-world applications.\n\nThere are many recent works (see below) in causal inference that try to combine multiple datasets. They are not directly very relevant to your paper. But your paper title and abstract are too general without specifying the problem setup you focus on. \n\nShu Yang, Peng Ding. Combining multiple observational data sources to estimate causal effects. https://arxiv.org/abs/1801.00802\n\nKatherine Evans, BaoLuo Sun, James Robins, Eric J. Tchetgen Tchetgen. Doubly Robust Regression Analysis for Data Fusion. https://arxiv.org/abs/1808.07309\n\nWenshuo Guo, Serena Wang, Peng Ding, Yixin Wang, Michael I. Jordan.\nMulti-Source Causal Inference Using Control Variates. https://arxiv.org/abs/2103.16689\n\nE Rosenman, AB Owen, M Baiocchi, H Banack. Propensity score methods for merging observational and experimental datasets. arXiv preprint arXiv:1804.07863\n\n\n\n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they do.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers advocate acceptance. I also recommend accepting the paper for its contributions to the emerging field of Bayesian causal inference.
Decision: Accept (Poster)