Efficient Policy Evaluation Across Multiple Different Experimental Datasets
Authors: Yonghan Jung
Alexis Bellot
Venue: NeurIPS 2024 poster (2024)
Keywords: dynamic treatment regime
data fusion
off-policy evaluation
transportability
Primary area: causal_inference
TL;DR: We provide estimators for evaluating the effect of policies leveraging data from related domains.

Abstract: Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from the same distribution. This assumption is often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability of the effectiveness of policies in a target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies (e.g., conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.

Efficient Policy Evaluation Across Multiple Different Experimental Datasets
Yonghan Jung"
Alexis Bellot Independent Researcher abellor att @mma 1 com
Purdue University jung222@purduee.co
Abstract
Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from same distribution. This assumption often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability the effectiveness of policies in target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies (e.g., conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.
Introduction
In the empirical sciences, conclusions on the effect of actions policies is often supported by evidence drawn from prior observations and experiments. The conditions under which such inferences can be formally justified can be traced back (in part) to Campbell, Stanley and Cook [10, 11, 14]. They argued for a basic dichotomy the kinds of questions that scientists seek to answer from experimental hand asking whether "in fact, the experimental stimulus made some significant
data. On the difference in this specific instance?". and on the other hand asking "to what populations, settings, and treatments this effect be generalized?" [10, p. 297]. These inferences have since been labelled as internal validity and external validity, respectively.
External validity is concerned with the extent to which findings from one population can be "re- processed", or "re-calibrated" so as to circumvent population differences and produce valid general- izations in target population where experiments cannot be performed (e.g., outside the laboratory, different domains, etc.). The validity of these inferences will necessarily be contingent on a careful analysis ascertain the commonalities and differences between domains as, for example, if the target domain is completely arbitrary generalization impossible. In the causal transportability literature, the basis for generalization (also called transportability) is justified by the stability and invariance of the causal mechanisms shared across populations and domains [20, 32). Several graphi- cal characterizations exist to delineate the conditions under which transportability is possible, with recent algorithms proposing solutions for general instances of the external validity task combining observational and experimental distributions under partial observability [36, 3, 16, 30].
$ Equal contribution. Correspondence Yonghan Jung: juwggg220uurreiieee edu. Now at Google DeepMind. Work done partly while affiliated with Columbia University and partly in an independent capacity.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

These algorithmic solutions express a target policy effect in terms of the observational and experi- mental source distributions. Still, then one needs to go further and estimate the resulting expression from finite samples. In practice, with finite number of samples and potentially high-dimensional covariates, estimating causal expressions
quite challenging. Effective estimators have been de- veloped for specific settings, starting with doubly-robust estimators for functionals given by the backdoor criterion [13, 37, 9, 45J, and recently extended to cover general identification scenarios with observational and experimental samples [25, 26, 8J. These techniques also find parallels across other related disciplines, such as reinforcement learning where re- re-weighting [42, 31J, outcome modelling [6]. and doubly-rouust estimation [18], are common for evaluating the effect of policies to overcome shifts in the behaviour policy. Recently. [44] and [22] have considered policy evaluation under covariate shift and selection bias, a special case of the external validity problem with given graph. Despite their generality, existing estimators still only cover a limited portion of realistic scientific inferences. In particular, existing methods are not applicable in settings where datasets are collected in different domains.
We consider the generalization of causal claims from observational and experimental data through the task of policy evaluation. The target for inference is Epg]Y] where PH symbolizes the distribution of data in a target domain (indexed 0) in which a hypothetical policy of interest (also known as dynamic treatment regimes [33] or soft interventions [16]) has been implemented. The question then becomes how to identify and estimate E [Y], given finite samples from multiple observational and experimental data (e.g., PH a source domain indexed by in which experimental policy is म, that may differ with To) collected under different settings and structural assumptions, encoded in causal diagrams. We aim to bridge the gap between identification and estimation to solve general instances of external validity. Our contributions are twofold:
1. Sec. 3: We develop nonparametric identification criteria (Thm. 1) to determine whether the effect of a policy may be expressed through adjustment formula from two separate distributions induced by policy interventions, collected from different populations. Based on this formulation, we develop a multiply robust estimator (Thm. 3) that enjoys multiply robustness against model misspecification and bias.
2. Sec. 4: We generalize these identification criteria (Thm. 4) and propose a general multiply-robust estimator (Thm. 6) applicable for the evaluation of policies from multiple source datasets.
1.1 Preliminaries
We use bold letters (X) to denote a random vector and X random value. Each random vector represented with a capital letter (X) and its realized value with a small letter (x). Given a set X- (Xi,... .x),, we denote ( == (X,,.... x,). For a discrete vector X, we use (X) to represent the indicator function such that (X) = 1 if X = x: [X) 0 otherwise. For comprehensibility, we use P(v) to denote a probability at V at v for discrete/continuous random variables V. In similar, we use Σx for Z s V for the sunmation/integraaion over mixture of discrete/continuous random variables For example, we write the back-door adjustment as Σ, Ey x. 2 P(z) even when Z is a mixture of discrete/continuous variables. We use [v[] = Ev f(v) for a function f. For a sample set D = (V() : : = 1, n) where V V(n denotes the ith samples, we use ED[s(V)] = (1/n) E (V()). We use [| == /EPL(((V)F If a function f is a consistent estimator of f having a rate The we use f-f- (r(rx). We say f Logconsistent if [l. flp (p().. We use - 0p(1) if f - ʃ is bounded in probability, and f f-f ff- Op(r,) when f - f is bounded in probability rate T..-
We use Structural Causal Models (SCMs) as our framework [35]. An SCM M is quadruple M = (U, P). U is set of latent variables following joint distribution P(U). V is a set of observable variables whose values are determined by functions F= (ʃv, : V.E E V) such that V, - Sv; (pay:: uv;) where PA; S V and Uy, S U. Each SCM M induces distribution
P(V) and a causal graph G in which directed edges from every variable PA, to V; exist. Dashed- bidirected arrows encode correlated latent variables.

---

2 Policy Evaluation Integrating Multiple Experimental Datasets
We investigate the sequential decision-making setting concerning a set of actions X, a series of dynamic covariates Z. series of static covariates C and an outcome variable of interest Y in an SCM M. A policy vector ग := { over actions X = {X1,.... , _m}
is an ordered set of decision rules for each X2 E X. Actions are selected according to a topological ordering X1 <.... < Xx over time. Each action X; is potentially associated with a set of prior static and dynamic covariates, for example, the decision rule for X2 could be defined as Tg ~ .(( 2 x(1),x((-1) c(*)). Every (Xxx ((k), X (k-1), C(k)) is a probability distribution mapping from domains of the set of inputs (Z(4) ((--1), 3 to the domain of actions Xg- The implementation of a policy æ in .M induces an intervened model M,. that sets values of every X X to be decided by the policy я,, replacing the functions {fx. X E ] that would normally set its value. We denote a distribution induced by Mr as Ps. Now, we fix the notion of the policy evaluation follows:
Definition (Policy evaluation [41]). The policy evaluation is to predict the effectiveness of policy vector π on an outcome Y in an targer SCM M:; i..., 1/- = [Y]
Difficulties in estimating Epe [Y] comes from that the distribution or samples from p0 are generally not available. These discrepancies can be formalized under the rubric of SCMs as follows. In the most general setting, an investigator might leverage multiple source domains (M', .M....., MM) over V that entail distributions IP: (P' p2 PK), }. Data or samples from these distributions may be available under different behaviour policies, e.g., T1, ....... TK. depending on the study or data collection protocol implemented in each domain (that might include an observational regime, i.e. no policy implemented). To ground the policy evaluation problem, we define graphical tools to capture commonalities and discrepancies across domains.
Definition 2 (Domain discrepancy [29]). For every pair of SCMs M (i,je E 10, 1,2,...... K)) defined over V, the domain discrepancy set Aij S V is defined such that for every V E Dij there might exist a discrepancy between JV * s or pas (uv) * pMM (uv).
Definition 3 (Selection diagram (291). The selection diagram 
(g (G*) )e(0.......... U 2e(1,......TT is a graph constructed from G (i (0, 1,2,.... ..... .T)) by adding the selection node Sij to the vertex set, and adding the edge Sij - V for every V E Aij.
Aij locates the mechanisms where structural discrepancies between two domains are suspected to take place. V & Ai,, represents the assumption that the mechanisms for V are invariant across the two domains. The induced selection diagram is a parsimonious representation of these constraints. The following example illustrates these notions.
Example (External validity under covariate shift). A common instance of the external validity problem in the literature considers the evaluation the effect of a policy Uo x Qx - [o, 1] for assigning a treatment X {0, 1), subject shift in the distribution of covariates C. For this example, let source and target domains MI (M3 follows.
over V {C, X, Y). U = {Uxy, Uc} be defined as
C SolUc)
C - 18(Uc)
M' F
X Fx(C,UxY) V- Fx(X,C,UXV)
M Fo
X - Fx(C,Uxx) Y CXU,)
P(U) P(UX))P(UCC
PO(U)) = P(UX))P((Uc)
Here, C' 11,0, (Y,C) $ A1.o as only the mechanism for C' varies across domains. Consider the evaluation of क T(X = 1 == + exp(-c)) -/(1+xxp(-c) given an experimental dataset in M in which X has been randomized, ie., X ~ Bern(0.5), and covariate data (C) available in M Notice that we do not have access to the specification of the SCMs M. but only the induced diagrams GA, and a subset of entailed distributions P. (Pinnd()) (X, (X,Y,)) Y,C), PC())). The policy effect is expressible as
E P
y ( (v)l c, ())(((
e) PC().
and estimated given the policy and the combination of the available data from P1 ppp
3

---

SGa
X1
X1
X
C2
X1
X2
C2
X1
C2
So
So
W
W
W
(a) g
(b) g
00.1 (c) gmp
(d) gro
Figure Graphs illustrating the inference of a two-stage treatment strategy no == (Tr (1), (z(22
21, C2, w)) given data from source domains M _M,, described in Example 2.
3 Combining experiments from two domains
Example illustrates two challenges in combining data from different domains to infer the effect of a new policy in a target domain. In first instance highlighting the challenge of identification, that is inferring an expression terms of IP that identifies the policy effect, and in a second instance highlighting the challenge of estimation, that is providing efficient estimators from finite samples for the identified policy effect. The following example will serve to motivate this setting.
Example 2 (Two-stage treatment strategies). A team of physicians is contemplating treatment plan To against heart disease Y for their patients in .M They consider administrating two drugs in sequence: drug against hypertension X1, followed by an anti-diabetic drug X2 depending on the effect of X1 on blood pressure W. To support their evaluation, two studies exist on these drugs, from domains M M2, that, however, have only analyzed their effect in isolation (on X, and X2 separately) and under different treatment guidelines, TT, T2 respectively. The data collected refers to the variables
(Y, C1, X1, X2, W) which (C,, C2) are demographic variables. Formally, we assume physicians have access to IP {PA, (V), P P C2)}. The superscripts in po Pl, p are the index for the domain, and the subscripts To, T1, T2 denote the policies for assigning treatments. G in Fig. 1 encodes the structural assumptions, which include discrepancies across domains and implemented policies in the available data. For example, the graph 9 specifies the known guideline π used in M while no specific plan was followed for the assignment of X2, that in practice depends on the patient's covariates C2 as well as unobserved factors, e.g. mood, health awareness, etc. (summarized in the bi-directed arc). In addition, selection diagrams describe differences between domains. For example, the edge {Sc, - C13 in - indicates a potential change in the distribution of covariates C₁ across domains M M The question then becomes how to estimate I. pe [Y] given (P),P).
3.1 Identification
Example 2 illustrates the complexity of drawing inferences from multiple datasets collected under different settings. We extend this example to provide general identification procedure for the effect of policies when two source datasets subject to different policies and/or discrepancies with the target domain available. Let V := (Y,C, X1, W, X2, W, Y,S) denote set of disjoint variables, where Y is an outcome variable, C, W) are covariates corresponding to two experiments, (X1, X2) treatment variables, and S denotes the selection nodes describing discrepancies across pairs of domains. Formally, the task signature is given as follows:
Input: Samples from P (P!, (V), P2 (V), (C C2)); structural assumptions g (Go G G2 A Ao,1 Goo.2.
Query: Estimate E Epg [Y] where p0 is distribution on the target domain and mo is target policy
assigning treatments with (X( C₁) and (o(X2 C2, W).
Given these inputs, sufficient condition for identifying the query is given follows:
Definition& 4 (Adjustment criterion for combining two experiments). Given G the adjustment criterion for combining two experimental datasets is defined by the following d-separation statements:
1. Domain transfer for Y: (Y 1L | C1 X2, W) in G
A0.2 ;; i.e, the distribution over Y is invariant between the source distribution from M and the target.

---

2. Domain transfer for W: (W IL S C1) in GRo i.e. the distribution over W is invariant between the source distribution from M and the target.
Adjustment for Y: (Y IL ne C2,C2, C2 X1, X2, W) in 9n, for {0, 2}; ie., the distribution over Y is invariant between regimes and TT2.
4. Adjustment for W: (W 11. TH C1 C2 X1) in 9x, for i 1); ie., the distribution over W is invariant between regimes To and #1.
The adjustment criterion could be shown to hold for Example 2. Specifically, domain transfer could be shown by inspecting Fig.5e as the set S (Sc, Sw is d-separated from Y. conditional on {C1, C2 X1, X2. W). Similarly, domain transfer for W holds as {Sc, Sc Sy is d-separated from W. given C1, C2 1 in Fig.5d. Similarly, one could verify the adjustment condition for Y by inspecting Fig.5c and the adjustment condition for W by inspecting Fig. 5a.
For this example, these conditions imply identifiability of the target query Epp [Y] given (gA,P))
Theorem 1 (Adjustment for combining two experiments). Under the adjustment criterion in Def. 4, the target query tho == EEpo [Y] is identifiable from the samples from Pa, (V), P (V), PD (Ci, C2). Specifically, it's expressed as follows?. 3:
po [Y] = Σ
Epg [Y] w. c. c, P (,, (w) c1. (c),
(1)
u.C.x
where X X2) and C (C1, C2).
Effectively, despite the differences across domains encoded in Example 2, the effect of the new combination of anti-diabetic and unti-hypertensive drugs mo, can be estimated using samples from experiments already conducted
and baseline characteristics of patients in M
3.2 Estimation
This section considers the estimation of the effect of policies, building on the identification criterion in Thm. 1. We first parameterize the identification estimand in Eq. (1) with two types of nuisance parameters and w. is a collection of regression parameters, and w is a collection of the ratio of distributions.
The regression nuisance parameters are defined as follows: 16 C, W, X) == Epz, [Y C, W
and AB(C W, Σn, m C, W, X1, 22) C1W. Recursively, #b(C, X1)
Epi [A3(C, W, X1) C, and id(C) E., (=) C). Eq. (1) can be param-
eterized as E Ep
[ E [pb
On the other hand, the ratio nuisance parameters wH: wit are defined as functionals satisfying the following properties:
Eg P = Ep
Ept (1)
(2)
A closed form of the w/ is provided in the later section at Eq. (14). By the definition of ratio nuisances,
Eq. (1) can be parameterized as Ep [Y] Epy, [ C, W. W.J. Equipped with these nuisances. we now present the DML-based estimator for the target query:
Definition 5 (DML for combining two experiments). Let D2 P2 (V), D -P,, (V) and
DO (C). Let L. 2 2 denote fixed number.
Sample split: For 1. L, randomly split D' D'r i (0,,,22 into L-fold. The t'th partition of the sample is denoted Dj. The complement is D' D'
Nuisance estimation: For each -,.... ,L. learn the estimator model 2& and pi for ph Hb using samples D2 D2 DD respectively. Also, learn the estimation model for W wif for wby wg using samples D' for i 0, 1,2, respectively.
'Thm. remains valid for mixture of discrete and continuous W. C. and X. For these cases, sums can be appropriately replaced by Lebesgue integrals. However, we continue to use summation notation in our explanation to keep the presentation of the identification result straightforward
5

---

3. Evaluation: The DML estimator d for Epg [Y] is then given as
L Σ
vb = L Bpy (8f - EDy En -111 EpplAt]
(3)
Estimating the ratio nuisance {a* [²) can challenging due to the necessity of estimating density
P
(C)
P. (W),,X))
ratios like
P
or
Pz (W/CCC. (.)). We employ the classification-based method for estimating the
P (C)
density [17, Sec. 5.4]. To illustrate this method, consider estimating P (C) We assign À
P* (C) P(A-1[C)
samples of C are from Pl, and À 0 if from P Then, it's provable that P72 (C) P(X=0(C)*
which can be estimated using off off-the-shelf probabilistic classification estimators.
The error of the DML estimator is presented below:
Theorem 2 (Learning Guarantees). Suppose ir. Bg < to and 17 < JO (C, W, (Y (C W,
w WC, (.)A - -p² X1)), and Φ (C: = W,);;, n w
Define (2 uz 2u,,,t x2)
vo. For 0,1,2, de- fine db 8² equipped with true nuisances (mh: Tél and oi as ø' equipped with estimated nuisances ibe Ri. Define Rg == (1/L) El (Exp [0]] Ep: [o(())) for 0, 1,2. Then,
1. The error v/o is decomposed as follows:
2 R4
$ - v0
Ep. Hil - HbHWg w/1l.
(4)
Let Ph.0 Vps [06]] With probability (W.P) greater than 6,
2
2
Pc.
2
10 2ill?
Ra
3
(5)
D
DDi
1=0
E=1==
3. Let 10.00 = Ep; [1gg]] $3. Let ((x) denote the standard normal CDE W.P greater than 1
v D
1
L2
L
2
Pr
0.4748M..0
pi
R4 < x
-(x)
(6)
Pk:o
2
D
DD.VDD
If the nuisance parameters jig and #f converge at rate of n 1/4 (where n is the size of the smallest sample set), the DML estimator achieves a faster convergence rate of n -1/2, This rapid convergence allows its asymptotic distribution to closely approximate the standard normal distribution, as is further clarified in the asymptotic analysis:
Theorem 3 (Asymptotic Error). Suppose each nuisance estimates Ptt D2, Wot are La-consistent and bounded. Then, the error of the DML estimator vp in Def. 5 is given as follows:
L
3-1 vi Σ
Σ t=1
Opg
C1Ap páll103 -wll + Σ Opg (l Hblll wil), (7)
where R2 converges in distribution to normal p2.
Eq. (7) implies that the error term d th converges to zero faster than the convergence rate of nuisances, which is property known as debiasedness.

---

W1
W
X1
(a) o
(b)
(c) g
8c.
SC
S6
W
Sws
SW
Sws
X1
X1
X
310
10
3o
(d) g
A..2
(1) Ao.a
Figure 2: Graphs illustrating the inference of a multiple treatment strategy To = { [c), m0(2
21; C22 W1), (T( 22, Ca, w2) given data from source domains M M M
4 Combining multiple experiments
In this section, we extend our method to incorporate the combination of data from multiple experi- ments, specifically focusing on different experiments derived from varied policies (π²² in distinct source domains (M'). A practical scenario for this task is the following:
Example 3 (Multi-stage treatment strategies). Consider a scenario involving hospitals in three different cities: New York (domain 1 with , G*), Los Angeles (domain 2 with P G², and San Francisco (domain 3 with p3, g)) Each hospital has different guidelines, i.e., policies, for diabetes treatment. In New York, the hospital focuses on insulin therapy adjustment based on the patient lifestyle choices, primarily for Type Diabetes patients (TT1 In Los Angeles, the hospital focuses on team diet and exercise regimen adjustments, primarily for Type 2 Diabetes patients (Λ22). In contrast, San Francisco's approach involves advanced monitoring and Al-driven predictive adjustments for higher-risk diabetes patients. Now, the leader of new clinical team in Chicago (the target domain with po, (D), the task is to evaluate a novel candidate treatment policy To- which integrates the strategies from these three domains Diabetes patients. The structure of the problem is captured in causal diagrams in Fig. 2, illustrating the gata-generaatigg process, the experiments in each city, and the assumed discrepancies between these source domains and Chicago.
provide comprehensive care for both Type and Type
4.1 Identification
We consider sequence of variables (C, X1, Wi.... Xm, W. ;= Y) where (C, W(i-1)) represent
the covariates corresponding to each of the i'th experiments, and (X1, Xm) are the corresponding treatment variables. We given samples drawn from P (V) for i 1,... mm and (C). We will leverage causal diagrams Gr, and selection diagrams Go.. for every 1,... m. Formally, the task signature is given as follows:
Input: Samples from Pi, (V) for 1,... m and (C C2); Causal diagrams G and
selection diagrams A for i 1,... m.
Query: Estimate the effect of the target policy To on the target domain M i.e. Epg [Y].
Definition 6 (Adjustment criterion for combining multiple experiments). The adjustment cri- terion for combining multiple policies are the following d-separation criterion in the the DTRs Do.m.
Gogg Gmys....
Ga.,. and the selection diagram G
Domain transfer for Y: (Y T1 S C. W.X) in Gāo.". "h; ; i.e. the distribution over Y is invariant between the source distribution from M and the target.
Domain transfer for W, for i 1.... m 1: (W; 11. ( W(i-1) in A iee, the distribution over W. is invariant between the source distribution from MD and the target.
7

---

3. Adjustment for Y: (Y 1L ns C. X) in In, for {0, m);; i.e,, the distribution over Y is invariant between regimes no and m-
Adjustment for W; i 1, m 1: (W; 11 J C (())) in Ox, for j E {0, i]; i.e. the distribution over W; is invariant between regimes To and nT.
These conditions lead to the following identification criterion.
Theorem 4 (Adjustment for combining multiple experiments). Under the adjustment criterion in Def. 6, the target query the == Ep [Y] is identifible from the samples from P (V), ........ pm (V)
and (C)) Specifically, it's expressed as follows:
m-1
mi-1
Epo M
Σ w,C,X
E Y
C. W
P (wi c, x(-(() w(i-
II j=1
To(az c, p(5-1))pp((a)
(8)
4.2 Estimation
The regression nuisance parameters are defined as follows. We first define the following nuisance.
Ho (C, X)X)= Epm
[Y C,W.
(9)
(C, W, x (m-1))
Σ (am C, W W( (C, W, v (m-1) xm)
(10)
For i = m - 1,... 1, the other nuisances are defined in a following manner:
ré(C w x()) = Epp [x, (C, w x(1) C, w x(),
(11)
(C, w(-1) x(i Σ Edl(C, w(i-1) ((i11) LT)TT( C, w
(12)
We note that Eq. (8) can be parameterized as Epp nuisance parameters w/ for i I, m are defined as functionals satisfying the followings:
y
Ep [ (C)]. On the other hand, the ratio
Epg [Y] = Epg Lnd(C, w x WWCC w ( (.
(13)
where the closed form is given as
w/ = To(X; C, W(-1))
pJ (W, C, C(-(1 W(--1)) C, P,
po C, ((i X(0)
(14)
Eq. (8) can be parameterized as Em (w) C, W(m-1), x (m)))). Equipped with these nuisances, we define corresponding estimator as follows.
Definition 7 (DML, for combining multiple experiments). Let D' Pd (V) for = 1,.... and D (C). Let 2 denote fixed number.
1. Sample split: For l 1,... randomly split D' for i 10, m) into L-fold. The E E'th partition of the sample is denoted Dic The complement is D'= Dij
2. Nuisance estimation: For each é 1,... learn the estimator model p fit for Mil ,14 using samples D DL., respectively. Also, learn the estimation model for w...... ,6o, for forwwy"" ".,"""""") using samples DD D'r 0, m, respectively.
3. Evaluation: The DML estimator a for Byg [Y] is then given
$ -= - Σ Σ Epylollit - pill + EDG [āb]]
(15)
The error of the DML estimator is presented below:

---

Theorem 5 (Learning Guarantees). Suppose Mo: fig < 00 and 0 < To. #t2 x almost surely for i= 1, Define (C, W x()); p jm+t Y. Let o" (C; == p¹ vo. For = , m. define oo as 6² equipped with true muisances, and o% as ø' equipped with estimated muisances. Define R₃ = (1/L) Ei-1
) w' -² for -.... m, where
Ep-lo/)) for 0, 1, 1. , Then, 1. The error dp vo is decomposed as follows:
m.
L Vpi [ob]] With probability (W.P) greater than
i - v0 = R4
Ep
Lik - MbH -a))]
(16)
Let 1ř.0
Pi.
I
P..
Σ 7-
R₂ < (m + 11
(17)
Di
[Dj
i-0
Let KN
Epp [p,,(000))) $1. Let (x)) denote the standard normal CDE W.P greater than 1 [1]
6,
\/ID
1
1
L
1
Pt
0.4748m330
p R4
((x))
&
(18)
Pk.o
V 2T
D
corresponding asymptotic error analysis is following:
Theorem 6 (Asymptotic Error). Suppose each nuisance estimates jul.... .. p and "i,".. ... 2.f are Lazconsistent and bounded. Then, the error of the DML estimator ub in Def. 7 given as follows:
$ -
= R
+
Σ
Op
(1A
milll w
(19)
T=U
E=1=1
where R₁ converges in distribution to No Normall (0, p3,0).
Similarly to Thm. this result implies that the DML estimator & converges fast even when the nuisance estimates converge relatively slowly.
5 Experiments
In this section, we demonstrate the proposed estimators in Defs. (5,7) for combining multiple experimental datasets from different domains. We first compared the estimators on synthetic data to provide evidence the fast convergence and doubly robustness behaviours of the proposed estimators. We conclude with analysis of the ACTG 175 clinical trial [21] and Project STAR. We will use (S)( for est (reg, pw, dml} to denote the estimators (OM, PW, DML.) for the policy effect pe Y. OM and PW estimators are purely based the regression based nuisances and w. respectively. To assess the quality of each estimator, we consider the absolute error (AB) as AB
(est x) Epo [- We used XGBoost [12] to estimate nuisances.
Synthetic Simulations We ran 100 simulations for each N (2500, 5000, 10000. 20000) where N is the sample size. We measure the AE the presence the 'converging noise in estimating the nuisance, decaying at N-1/4 rate (i.e., normal( - 1/4 N --1/4), where N the size of samples). To enforce the convergence rate of nuisance estimates faster than the decaying rate n 11/4, we add to all nuisance estimates. This scenario is inspired by the experimental design discussed in [27]. The AE plots for combining two/multiple experiments are presented Figs. (3a, 3b). For all examples, the proposed DML. estimator outperforms the other two estimators by achieving fast convergence. This result corroborates the robustness property Thm. (3, 6), which implies that the proposed estimator converges faster than the other counterparts.

---

0.6
0.25
0.20
0.4
0.5
0.3
0.10
0.1
0.3
1000
0.05
6000
10000
20000
5000
19000
20000
502
1000
2000
1000
2000
Semple now (4)
Semple size (x)
Samplesion (x)
Sample (n)
(a)
(b)
(c)
Comparison of the proposed DML estimator with other counterparts (outcome based model called 'OM', and the weighhig- model labelled 'PW") for (a,b) synthetic data analysis for combining and multiple experiments; and (c,d) real-world data analysis under the noise-frec or noisy environments learning muisances.
Figure 3:
To provide empirical evidence, we analyze the ACTG 175 random- ized trial [21], which assessed therapies for reducing CD4 T cell counts in HIV patients. Participants were randomly assigned to treatments X2 E {0. 1), with prior anti-retroviral drug use X₁ E {0. recorded. Patient demographies C₁, C2-including gender, age, weight, and Karnofsky score-were collected, and CD4 T cell counts (W) were measured. To simulate an alternative study with a modi- fied guideline for X1, we sub-sampled ACTG 175, adjusting covariate distributions and assignments of {X1, X2). Specifically, we evaluate a stochastic policy To = { c1). To c2)] for combining X1 and Xa based on C1, C2, with distribution fa representing a location with differing covariate distributions and treatment assignments. Further details are provided in Appendix D.2.
External validity: ACTG 175
We evaluated the AEst of all proposed estimators with and without noise (as described in the synthetic simulations). The AE plots are shown in Figs. (3c, 3d). Results indicate that both the regression and DML estimators converge to the true policy effect faster under noisy conditions, whereas the PW estimator converges more slowly. However, DML does not consistently outperform at all sample sizes (see Fig. 3c), as its error is influenced by the combined errors in the OM and PW estimators. Consequently, high error in the PW estimator may lead to increased error in the DML estimator.
External validity: Project STAR We further examine policies on teacher-student ratios (i.e., class sizes) to improve academic achieve- ment, using semi-synthetic adaptation of the Project STAR dataset [40]. This longitudinal study evaluated the impact of teacher-student ratios academic outcomes for students in kindergarten through third grade, with students randomized each year to one of three class size interventions. Here, we assess 3-stage policy setting student-teacher ratios across Grades 0, 1, and 2, observing academic scores as interme- diate outcomes, with baseline covariates (e.g., ethnicity, gender) and final academic scores at the end of Grade 3 as the primary outcome. To emulate data collected across different domains, we subsample using various probabilities shift baseline covariate distributions, as done in ACTG 175 (see Appendix D.3 for details). We evaluated the PW, OM, and DML estimators across dataset sizes, plotting their absolute errors against the true effect of the candidate policy. Results, shown in Fig. 4. mirror earlier experiments, with all estimators improving as sample size increases and DML showing faster convergence.
0.5
0.4
4 0.3
0.1
0.0
6000
10200
Sample size (n)
Figure 4: STAR Results.
6 Conclusion
This paper has considered the evaluation of the effectiveness of policies in settings where the available data is sampled from distributions that differ from the population in the target domain. We have illustrated this task with the problem of extrapolating the results of a clinical trial in both working examples and real-world scenarios to evaluate variations of the treatment different populations. Our contributions are (1) introducing several identification criteria for the effectiveness of policies given experimental datasets from two or more domains and (2) developing doubly robust estimators for these settings that achieve fast convergence.
10

---

Acknowledgments
We thank anonymous reviewers for constructive comments improve the manuscript.
References
[1] Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pages 687-696, 2017.
[2] Elias Bareinboim, Sanghack Lee, Vasant Honavar, and Judea Pearl. Transportability from multiple environments with limited experiments. Advances in Neural Information Processing Systems, 26, 2013.
[3] Elias Bareinboim and Judea Pearl. Transportability from multiple environments with limited experiments: Completeness results. Advances in neural information processing systems, 27, 2014.
[4] Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27:7345-7352 2016.
[5] Alexis Bellot, Alan Malek, and Silvia Chiappa. Trraspootabbility for bandits with data from different environments. Advances in Neural Information Processing Systems, 36, 2024.
[6] Andrew Bennett and Nathan Kallus. Efficient policy learning from surrogate-loss classification reductions. In International conference on machine learning, pages 788-798. PMLR, 2020.
[7] Andrew C Berry. The accuracy of the gaussian approximation to the sum of independent variates. Transactions of the american mathematical society, 1921):122-136, 1941.
[8] Rohit Bhattacharya, Razieh Nabi, and Ilya Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. The Journal of Machine Learning Research, (3(1):13325-13400 2022.
[9] Peter Bickel, Chris AJ Klaassen, Peter Bickel, Ya' acov Ritov, J Klaassen, Jon A Wellner, and YA' Acov Ritov. Efficient and adaptive estimation for semiparametric models, volume 4. Springer, 1993.
[10] Donald Campbell. Factors relevant the validity of experiments in social settings. Sociologi- cal methods, pages 243-2633 2017.
[11] Donald T Campbell and Julian C Stanley. Experimental and quasi-experimental designs for research. Ravenio books, 2015.
[12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[13] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
[14] Thomas Cook and Donald T Campbell. Causal inference and the language of experimentation. Quasi-experimmntttion:: Design & analysis issues for field seftings, pages 1-36, 1979.
[15] Juan Correa and Elias Bareinboim. General transportabilliyy of soft interventions: Completeness results. Advances in Neural Information Processing Systems, 13:10922- 2020.
[16] Juan D Correa and Elias Bareinboim. From statistical transportability to estimating the effect of stochastic interventions. In ISCAI, pages 1661-1667, 2019.
[17] Iván Díaz, Nicholas Williams, Katherine I. Hoffman, and Edward J Schenck. Nonparametric causal effects based on longitudinal modified treatment policies. Journal of the American Statistical Association, 1166542) 2023.

---

[18] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.
[19] Carl-Gustav Esseen. On the liapunov limit error in the theory of probability. Ark. Mat. Astr. Fys., 28:1-19, 1942.
[20] Stuart S Glennan. Mechanisms and the nature of causation. Erkenntnis, 44(1):49-71, 1996.
[21] Scott M Hammer, David Katzenstein, Michael D Hughes, Holly Gundacker, Robert Schooley, Richard H Haubrich, W Keith Henry, Michael M Lederman, John P Phair, Manette Niu, et al. A trial comparing nucleoside monotherapy with combination therapy in hiv-infected adults with cd4 cell counts from 200 to 500 per cubic millimeter. New England Journal of Medicine, 335( 335(15):1081-109) 1081-1090, 1996.
[22] Tobias Hatt, Daniel Tschernutter, and Stefan Feuerriegel. Generalizing off-policy learning under sample selection bias. In Uncertainty in Artificial Intelligence, pages 769-779. PMLR, 2022.
[23] Li He, Long Xia, Wei Zeng, Zhi-Ming Ma, Yihong Zhao, and Dawei Yin. Off-policy learning for multiple loggers. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1 1184-1193, 2019.
[24] Yonghan Jung, Iván Díaz, Jin Tian, and Elias Bareinboim. Estimating causal effects identifi- able from combination of observations and experiments. Advances in Neural Information Processing Systems, 36, 2024.
[25] Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects on markov cquivalence class through double machine learning. In International Conference on Machine Learning, pages 9168-5179 PMLR, 2021.
[26] Yonghan Jung, Jin Tian, and Elias Bareinboim. multiple experiments. In International Conference on Machine Learning, pages 15451-15527. PMLR, 2023.
Estimating joint treatment effects by combining
[27] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008-3049, 2023.
[28] Edward H Kennedy, Sivaraman Balakrishnan, Max G' GeSlll et al. Sharp instruments for classify- ing compliers and generalizing causal effects. Annals of Statistics, 48(4):2008 2030, 2020.
[29] Sanghack Lee, Juan Correa, and Elias Barcinboim. General suusportablliyy-yyteeeshit observations and experiments from heterogeneous domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10210-10217, 2020.
[30] Sanghack Lee, Juan D Correa, and Elias Barcinboim. Generalized transportability: Synthesis of experiments from heterogeneous domains. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020.
[31] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review. and Perspectives on Open Problems, 5, 2020.
[32] Peter Machamer, Lindley Darden, and Carl F Craver. Thinking about mechanisms. Philosophy of science, 67(1):1-25, 2000.
[33] Susan A Murphy. An experimental design for the development of adaptive treatment strategies. Statistics in medicine, 24(10): 1555-141,, 2005.
[34] Sonali Parbhoo, Shalmali Joshi, and Finale Doshi-Velez. Generalizing off off-policy evaluation from a causal perspective for sequential decision-making arXiv preprint arXiv:2201.08662, 2022.
[35] Judea Pearl. Causality. Cambridge university press, 2009.
[36] Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. In Twenty-fifth AAAI conference on artificial intelligence, 2011.
12

---

[37] Andrea Rotnitzky, James Robins, and Lucia Babino. On the multiply robust estimation of the mean of the g-functional. arXiv preprint arXiv:1705.0858/ 2017.
[38] IG Shevtsova. On the absolute constants in the berry-esseen-type inequalities. In Doklady Mathemalics, volume 89, pages 378-381. Springer, 2014.
[39] Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confi- dence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119(545):273-255 2024.
[40] James H Stock and Mark W Watson. Introduction to econometrics. Pearson, 2020.
[41] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[42] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems, 28, 2015.
[43] Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 10276-10283, 2020.
[44] Masatoshi Uchara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under covariate shift. Advances Neural Information Processing Systems, 33:49-61, 2020.
[45] Mark J van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal effects of multiple time point interventions. The international journal of biostatistics, 8(1), 2012.
[46] Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-arme bandit: causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 1778-1780, 2017.
13

---

Supplement to "Efficient Policy Evaluation Across Multiple Different Experimental Datasets"
Contents
1
Introduction
1.1 Preliminaries
2
Policy Evaluation Integrating Multiple Experimental Datasets
3
Combining experiments from two domains
3.1
Identification
3.2
Estimation
4
Combining multiple experiments
4.1
Identification
4.2
Estimation
5
Experiments
6
Conclusion
10
Related Work
15
Broader Impact Statement
15
Proofs
16
C.1
Proof for Theorem and Theorem
16
C.2 Proof for Theorem 2 and Theorem 5
16
C.2.1 Proof of Mixed Bias Property
16
C.2.2 Proof for Statement 1
17
C.2.3 Proof for Statement 2
18
C.2.4 Proof for Statement 3
20
C.3 Proof for Theorem 3 and Theorem 6
21
Details of Simulations
D.I Data Generating Process for Synthetic Simulations
Sime:a..........................
21
D.1.1
Synthetic Simulations for Fig. 3a
................................
21
D.1.2 Synthetic Simulations for Fig, 3b
22
D.2 External validity of the ACTG 175 clinical trial
22
D.3 External validity of the Project STAR study
23
14

---

A Related Work
Evaluating the impact of a policy using observational and experimental data under different condi- tions is widespread challenge in various important decision -making fields. Formulations of this problem have appeared in the causality literature, but also statistics, reinforcement learning, and epidemiology.
Off-policy evaluation (OPE) aims to assess the performance of policies of interest using observational samples. In this line research, [44] considers generalizing the effect of policy under distribution shift. We build this intuition, but instead seek combine multiple from source domains to learn the effect of policies of interest on the target domain. [1, 23] for instance used auxiliary datasets from multiple bandit instance, though they setting assume that the datasets are sampled from the same underlying populations and environments. Several authors have also considered transfer learning in off-policy learning in the context of bandits [46, 5J. Further, [22] addresses the problem of selection biases observational data for off-policy learning.
iittrvvtintr data
In the causal inference literature, combining multiple experimental studies to estimate new causal effect is a task called generalized identification [30]. Recent progress has been made in developing corresponding estimators [26, 24). However, these estimators are not applicable when our goal is to combine multiple policy interventional studies from source domains estimate a causal effect in the target domain. Accordingly, [4, 29, 15] developed the notion of generalized transportability that aims to evaluate a causal effect on a target domain from multiple observational and or interventional distributions from other source domains. In this line of research, our work relates closely Correa and Bareinboim's 's identification algorithm for the effect of policies [15]. We similarly develop identification criteria that are conducive to efficient estimation from finite samples. In particular, our work focuses on the derivation of sample-efficient estimators for the policy effect of interest on the target domain.
From this perspective, our work can be interpreted as a bridge between causal inference and off off-cy evaluation [34] since we leverage formal theories in causal inference (e.g., generalized identification [29]. generalized transportability [15, 30]) to solve off-policy evaluation problems efficiently from finite samples. There are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature use the backdoor adjustment to learn the Q value as a function of the state, to address confounding effects [41]. Mcanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders [39]. Finally, some works have leveraged double negative controls for OPE [43].
B Broader Impact Statement
Our work investigates the conditions under which policies may be estimated from multipe datasets collected under different conditions. In this work, we start from the assumption that causal and selection diagrams that are consistent with the underlying data generating systems interest are available. In general, this requires domain knowledge and should be justified by prior knowledge or experiment. It is important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation, that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns mostly systems, the computational complexity of estimating the
the second task. In higher-dimensionn conditional expectations and density ratios that define our estimators could be substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision decision-making process.
15

---

C
Proofs
C.1
Proof for Theorem and Theorem
Since Theorem is special case for Theorem 4, we will only prove for Theorem 4.
Note
Pgg Y
= Σ
Eo Y c, w.
II
w(i p (w) c, (
II ==1
To(Øz
c, (_w
70
W. W.C,X
1=1
(1)
Then,
Epe ag
[Y c, w,
Erg [Y
c, x|
(2)
= Epa Y w x],
(3)
by leveraging the domain transfer for Y and adjustment for Y condition. For each Po (w:/ c, (i-1), w(i
po mo (wi c, (ii (ii-)))
(4)
Pt (wi c, X w((-1))
(5)
P (wi c, (i-1)) w ((i-))).
(6)
again, by leveraging the domain transfer condition for W; and adjustment condition for W. This completes the proof.
C.2 Proof for Theorem and Theorem
Since Theorem 2 is special case for Theorem 5, will only prove for Theorem 5. Throughout
the proof, we will use C₁ :- C, X, := (X,) for 1. m. and C, := (W-1) for =
2, ... mm 1. Also, we will sometimes use PC(CC) (C), PC(C, ((-l)) x(( for
P-l (W,-1 C, w ()-))).
C.2.1 Proof of Mixed Bias Property
Using the fact that vh = po [ill we can write it as
vo =
E
[ool
+
Ep
=
> Eps [o
(7)
Then, we will claim and prove the following:
Lemma 1 (Mixed Bias Property). Suppose 1b f < oc and 0 < Té. ## < almost surely for
i= 1.... For 1,2, ,,, define
d'((C, W X ):w,, j ,1²) w ( 11))
8
and pm=+1 = Y, Define (C; (C:p))== = p.. For i =, 0,... m, define oi as oi equipped with true nuisances, and & as of equipped with estimated muisances. Then,
m
m
Σ
Ep
oil Σ
Eps H1' Holtub w61l.
Proof of Lemma 1. For i m,... with m=+ = Y, define
[ = Eyg [z C, w x(]
16

---

Then.
E mm
(
11 + Epm 1r
Epm
[w]) 140" [at 11=
mm
Vo
Ep 11000 wp" M4O [1 3..
For i = m
1, 1.
Ep [6 (1 - [x-)] + I2 [wpn(]] - Eps, [
= Ep; M100 wit ( - f2'11-
Also, the following holds:
E p:ll (wb w aottatt]] Erg (i
Finally, Epg, [
= Epp Ja
Therefore.
E pm 1 (1ttttt +1 - - Brg [Wfa"] Erg (
Epp 100- - ob1l
> Ep M1:0 #6 11
C.2.2
Proof for Statement
Recall
1 $6 =
Σ
- p81l + Exylüthl
=
1 Σ
E
E
[d
(10)
L
L 2=1:00
1=
From Eq. (7),
1
Vo :
E [ob]]
(11)
L
Then, the error voo can be decomposed into
ED pt lobl
L Σ Σ
Ep- P.
Ep.
[0i
261 +
Σ
Epg
E
[0]
-66 (12)
L
1-0
t= 1=
Define
1
R, == EDi-p
[16]]]
+
Σ
E
[0 16]
I
1-1
Then, the error can be represented as
$ -
Σ
Rs +
Σ
E 10 Hi
T.
1=0
f-1
1=0
17

---

By Lemma 1,
1 L
16
40
R4
E Brg [ woHu -m
1-0
2-1
C.2.3 Proof for Statement
We will use the following results:
Lemma 2 (Combining concentration inequalities). Suppose P(Ag > t) & bk/t fork 1,... K.
Then,
K
K
A S tk
>1
bk
Proof of Lemma 2. The event Ri-1 Therefore,
A₂ & tk includes the case where Ag < for k: 1,.,, K.
K
P
Σ
A $ tk
> P(A1 & and and Ax < t)
1 P(A > for or Ak > 1)
K
3
Σ
((A+
K
by
12
Lemma (Stochastic Equicontinuity). Let D did P. Let D Do Dj, where |ool Let f be function estimated from Di, Then, in probability greater than for any (0,
ED, p [[-- 5| up 1f fl
[
(13)
which implies that
(1f-flle.
|l vn
ED.- - fl] = =Op
Proof of Lemma 3. This proof is from [28, Lemma 2]. Since f is a function of Dj, we will denote
fD. Define following random variable of interest:
= ED.-P(fr, - -.]
Then, the conditional expectation of X given D1 is zero, since
Ep 1 (o.((V)) (V.)
Di = 1 Σ Eplie, D. = 1 - Erl,,, (V) D2] Erlfe, Pt).
Dı
where the third equality holds by the independence of DD and Dj. Therefore,
Ep[X] Di] Ep[E2, 1iEE,,-r(fer rLfo, - Di]
Er(ED, ITD, - f1 Di]] Er[Er[Jr, |
E.[EE[fo, fl Di1 Er(Epffo - -1] = 0.
18

---

Also,
Wp Dil VrIED, - - Al D:]
VrIEn Di]
Velfo, - D1)
n f1??.
1 fl.. f1l.
By applying the (conditional-) Chevyshev's s inequality,
1
P( - Ep[x Dill > t Di) & p Vplx Dil & nt² nellfo.
llfo. - -flls.
Then,
P( 3 = P(IX - Bp[x Di]l > 1) EProo) - Ep(X Dil t Dz)l
//2 nt
1 lifo flp..
In other words, X < t in probability greater than
nra 1ls, - S11.. If t.
1f0, -s1 then Vne
-fee in the probability greater than 1 - for any ce (0. 1).
X
//c
Here, will study the finite sample behavior of
Σ
m
R4
m ED pi
[06]
1
Σ
m
Edi P. /ot [b
L
1-
t-11-0
By Chevyshev's inequality,
Pr (Bp- [0b]]
Ps.0
1
v |Di
72
or equivalently,
Pr
EED (ED [bbl > >t)
t² DDi
By Lemma 2,
172
m
2
Pr
E E p. [oil & (m + 1)th
21 1
Σ
t
Di
0-0
By Lemma 3,
11 P1 Pi
Pr BBD m (02 Ub] > 12) < 12
DDj
By Lemma 2,
1b1l P.
P
1
Σ
Σ
Epj pi. 1o] - Vbll (m + 1)(2
2 1
T,
[D]
2 1=
19

---

1
Choose 11 = V
mm 2i-0
pLa Da
and l2 == V {
pt. D21
Then, with probability greater
1-0
than -
Σ 1=0
m
R4 < (m + (m+1)
2
Pi.0
2 K
k 2 DDj
pc
Di
1=0
8=
i=1
Bbl
2
= (m + 1)1
Σ
+
Di
IDjl
1=1
C.2.4 Proof for Statement
We will use the following result:
Proposition 1 (Berry-Esseen's inequality [7, 19, 38)). Suppose D {X,,... _n)3 in-
dependent and identically distributed random variables with Ep [X]] = 0, Ep[x?] = o² and [r[[x,]] x3. Then, for all x and
v
T
0.4748x
p
[x[x] <x) 0(x)) & 03.7
00
By Lemma 3,
11
6 P
Pr
(BDj-Pt 10
4/1
>t)
1
(14)
22
D
By Lemma 2,
10
o 12
1
1
P
Pr
ED; ps. 10
>
1
(15)
22
D
8=1
Define
81 A
A, :=
DDi
With a probability greater than 1-6, E,
wpl-c
ED ps [0] Øil
$ A.
L
Define
A; == -p:
Ep-
1
B :
Eypy Bopy-
Pi [00 - Φb]
L 1-1
C = T,
7,
1
Σ
BDD; - pi 10] (02-08)) obll
8-1
20

---

Then. R² = A; + B,. Then,
Pr Pr <))
(16)
= Pr + Bi <x)
(17)
PA <I-
(18)
< Pr(AA <x+
(19)
"pl-e Pr (r <x+A)).
(20)
Then,
- ((x)
(21)
(Pr 0 A,) + + + A ((x)
(22)
(Pr(A <x+ 0( A.) + |0 + 4,) +(x)
(23)
0.4748k
+10(1 A.) {()]
(Prop. 1)
(24)
PloVID
0.4748k
(
(Mean-vate value theorem)
(25)
PloVID4
0.4748k
1
(26)
22
This completes the proof.
C.3 Proof for Theorem 3 and Theorem
By Cauchy- Cauchy-Schwartz' inequality,
1
Bpg E=
Bp HAb AMO
m
will
1 L m
Σ > Opt (leb A Pull afll)
(27)
L
L
[=1==0
Given assumption, the upper bound in Eq. (18) converges at 1/ //D/ rate. Therefore, R; converges in distribution normal(O,, (.,))
D Details of Simulations
D.1 Data Generating Process for Synthetic Simulations
Codes corresponding to simulations are submitted as supplementary materials.
D.1.1
Synthetic Simulations for Fig. 3a
We define the following SCM. First, Uxw, Ux, UN,,UX,,,,UU... Uc,, Uw, Uy normal(0, 1). Then,
= fc 0.25SU 0.15 Uc..
fc, +
=1 [1,xx(( Bernoul((a,s(CI.Cc))
: I,, X₂ = (X, C1 S2 Bernoul((r,s(CI.CG))
+..50 3X, + 0.5(CI + C2))
Y Sy X,, X2, Y sigmoid(05(()++C)) C2) + 2(X; + X²) 0.5W
+ +
21

---

Also, for S # 0,
01,0 = s
-2-2)
T15 = s i (0 (2) + -1)
220 = s 1igmoid(0.((C + + 0.5W + +1)
22.5 sigmoid C2 + 2X, - -1 - 1).
D.1.2 Synthetic Simulations for Fig. 3b
We define the following SCM. First, Ux,,X Xx UW, X,: Uw, X2, Uw Uw, Xs.Uc.. Ua..2 U, U Uc,,, UC,,, Uc
U. UC,, U, Uw, Uw,, Uy ~ Normal(0) 1). Then,
C1 = fc, (9) = 0.255Uc, + 0.15 + U + Uc..c
C2 fc,
0.25SUc,, 0.18 + U22, Uc,e, + Uc2.0
C3 == fc; (S) 0.255U + 0.15 + Ucs.2 Uca.0s
X1 (C1, C2, 9) Bernouli(m.s(C,,C))
Wi = (V, U,, sigmoid(0225SUw, 0.5(C1 C2 + Ca)
+ Uw..22) + +S)
x= fx,,, C1, C2, Ca, Bernoul(fr2s(X,,,,,,C C2, Cs S))
Wa fw. (C, Wi, UW, U 8) sigmo
C2 + Ca)
+ Xx) + Ug + S)
X3 fx, X2, W1, W2, Ci Ca, Bernoul1(f2.s(X..kg, W2, W. Cr, C2, C2, S))
y= (C, UX, signooc((00((1++++++ 2(x; X₂ + Xa)
-3
0.255U +-500++55) +S).
Also, for S 36 0,
T1.0 sigmoid(C+++++G4--
T1.S +igmood(00((11+++++++
2200 = s 3moo:d(0ss((+++++++++++++++ 1) 0.5W +1)
T2.5 si 4gmosa(((++000002222-1-1-5
W3.0 si c
- -12-0. +1+ (2X, - 0.25W, + +1)
T3.S = s 11gmoid(05( C2 +
+ 2(2X; - -1) + 0.25W, - 11 2222 - 1) + 0.25W, 1).
D.2 External validity of the ACTG 175 clinical trial
To provide empirical evidence of policy estimation in real-world setting, we revisit the ACTG 175 randomized clinical trial from 1994 conducted on patients from the United States and Puerto Rico [21]. It investigated the effectiveness of different therapies for reducing CD4 T cell counts in individuals with HIV (selected subject to various inclusion criteria). the study, individuals were randomly assigned two different treatments X₂ {0, 1), and record
made whether previous reti-retroviral drug had been administered X₁ E {0, 1} prior to the start of the trial. Patient demographics C₁ C2 including gender, age, weight, among others, were collected, and CD4 T cell count were measured at treatment time W, and again 20 weeks after treatment initialization Y, the outcome of the analysis. To simulate second study with a different guideline for anti-retroviral drug administration, we considered a sub-sampled version of ACTG 175 in which covariate distributions as well as the assignment of X,, X2 were modified.
ACTG 175 is an experimental study in which X₂ has been randomized and X1 follows a base- line, unknown, stochastic policy 72 : Scc x f2x, - [o, 1] assumed to depend on study-specific features C2 such patient's Karnofsky score and symptomatic indicators (both normalized to lie in the [0. 1] interval). Samples of variables C1, C2, W, X2, Y therefore follow distribution rama( x2) (C1, C2, W,X,, X2, Y). The suffix (Xa))(X2) denotes a policy that randomizes X2, i.c. X2 (0rm(..5).
22

---

C2
C2
C
- W
C
(a) GATTO
(b) g
(c) g
T0
50
Sca
50
50
Sc
X
Sc:
C1
W
C1
W
(d) GTo
(e) gmo
So.2
Figure 5: Causal diagrams and selection diagrams of the ACTG 175 experiment.
We generate data sample from second domain (S 1) following the marginalized distribution Pl (C1, W, X1), mimicking simple stochastic guideline on X1 in which тī := Ti Ti(x = |c)) = 1/(1 + cxpf-c11 C2 Cia)), ). Higher values of C₁ (taken to be normalized measurements of weight, height, and age) lead higher likelihood of treatment. We achieve this dataset by sampling according to re-weighted version of the ACTG 175 trial. In particular, we collect data from pl according to,
#. (C1, W,) = P2 (C1, W, T1 ()P P (X1 C1)
For this example, consider evaluating a stochastic policy mo ( (i), <0 (a)) that combines the drugs X2 according to a stochastic policy for X₁ based on weight, height, and age, (Ci) and for X2 based on a patient's Karnofsky score and symptomatic indicators (C2). In particular,
To(2 11 cI) 1/(1 exp( C11 13), no(22 1 (2) 1/(L + -x5p2-05522 - c22)). (1)
The policy To is considered to be implemented patient population located in a different location that are know to have differing covariate distribution (C1, C2) to that observed ACTG 175 and the second study, among other discrepancies. We assume that the SCM generating this experimental study follows the causal graphs in Fig. 5.
The target population under no is then given by
pd (C1, C22 W. X1, Y)
CC C2) (2 P2 (Xi C2) (and (x)) (X2)
C2)
P (2) (C1, C2, X,,
We limit all datasets to approximately 2000 samples as this is the size of the ACTG 175 trial. The ground truth target effect po [Y] is evaluated by taking the empirical mean of in the sample of data collected from p0 with the procedure above.
D.3 External validity of the Project STAR study
We describe in this section additional experimental details the Project STAR study+ This study investigated the impact of teacher/student ratios on academic achievement for kindergarten through third-grade students. Project STAR was a four-year longitudinal study where students were randomly assigned to one of three interventions with different class sizes each year, following different randomization procedures. The causal diagram we assume for this setting is provided in
'The dataset is publicly accessible httpss:/search..-ppe htteer//eacca...rooootttt//////e
from the R data repository:
STAR html.
23

---

Fig. 6. Bi-directed ares denote unobserved confounding in the observational regime (when student are observed in particular class size rather than forced to join a particular class size).
Specifically, we consider the evaluation of 3-stage stochastic policy,
To {mo (1), To (22 22, 21, w1), mo(23 c2. T2, w2))
where,
to G1) == 1/(1+ c12 2]) To(22 02, w1) : 1/(1 + exp(0.5(c21 + 022) + 2(2x1 1) 0.5w, + 13)
C3. T2, W2) == 1/(1 + exp(0.5((c + c22) + (2x1 - 1) - 0.25w, + (2x2 1) - 0.25uz + 1))
These policies determine the student-teacher ratio Xo. X1, X2, taking values "regular" or " "small", across three different grades, namely Grade (Kindergarten), Grade and Grade 2. C refers to wwo-dimensional demographic variable encoding gender and ethnicity, converted to binary and categorical variables respectively. (To avoid clutter, in Fig. 6 we use C = C₁ C2 C3.) W1, W2 are intermediate school outcomes that include the sum total of an individual's reading score and math score in grades 0 (Kindergarten) and respectively. Y is the outcome of interest and represents total reading score and math score in grade 2.
To mimic the setting where data at different stages was collected from different domains, we subsample the dataset using different sets of probabilities to induce differences in the distributions of baseline covariates. In particular, we fix the dataset in the target domain (S = 0) to the distribution observed in the study and sub-sample according to different probabilities create datasets for domains S S, = 2, and S = 3, as follows.
We generate a sample of data from a first source domain (S = 1) following the marginalized
distribution P1 (C1, X1), where क == (1{ 1 [1) 1/(1 + exp(0.5(c11 c12) - 1)) defines the probability for the student-teacher ratio variables in Kindergarten in domain S = 1. We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from pl according to,
Pt (C,, W1, X1) = (C, X1) (X, (C))
(X, C1)
where P' (c1) = 0.3 if c11 1,2 = 1 and P' (c2) 0.7 otherwise.
We generate a sample of data from a second source domain (S = 2) following the marginalized dis-
tribution p2 (O2, W1, W2, X1, X2),, where T2 (2 1 02 31, w1) 1/(1 + exp(0.5(c2) + (22) + 2x 1- - 0.5w] - 1}) defines the probability for the student-teacher ratio variables in grade 1 in domain S = 2. We achieve this dataset by sampling according to re-weighted version of the
STAR study. In particular, we collect data from p2 according to,
T2 (2(22 C2. X1, W))
p2 p (C,, W1, W2, X1, X2) pd (C2, W1, We, X1, X2)
(X2 C2, Wi)
where p2( (c2) 0.7 if (2) 022 = 1 and (c2( = 0.3 otherwise.
We generate sample of data from third source domain (S 3) following the marginalized
distribution ق3 (Cs, W, 22, X1, X2) where F3 = T3 T = 1 C32 I1, W1, X2, w2) = 1/(1+ + exp(0.5(c3) + 632) + 2(xx - 1) - 0.25w: + (2xr² - 1) + 0.25w² 1}) defines the probability for the student-teacher ratio variables in grade in domain S 3. We achieve this dataset by sampling according to re-weighted version of the STAR study. In particular, we collect data from p3 according to,
(C,, Wi, W2, X1, X2, Xa, Y) =
(C, W1, W2, X1, X3, V) Ta (Xa Ca, Wi, W2, X1, X2, (z()) P C3, X1, W1, X2, W₂)
where (ca)
0.5 if C31
1, 1,032 1 and (3 0.5 otherwise.
24

---

C1
C2
C3
X1
X2
X3
W2
Figure 6: Causal diagram assumed for the target domain of the STAR Project study. To avoid cluttering the diagram write C Ci Cs, i.c., all C's refer the same variables (gender and ethnicity).
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA]
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short sentence) justificctinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] generally preferable to "[No] ", is perfectly acceptable to answer "[No] providedd proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
reported because it would be too computationally
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made match the theoretical and experimental results presented in the paper. A broader overview statement in the Appendix reflects how much the results can be expected to generalize to other settings.
Guidelines:
25

---

The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a broader overview statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability in real-world settings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical statement are quoted in full in the paper. We have attempted provide an example to illustrate the significance of each theoretical statement and highlight its implications. The formal proof of all statements given in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
26

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide the data generating mechanisms, details of the target of estimation and information as to what python libraries can be used to the proposed estimators. We do not, however, disclose an open source implementation of the proposed methods this moment.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The data is publicly available and we have provided full details as to where to access the data and how to run the synthetic data generation pipeline. The code will not be open sourced at this moment but believe to have provided sufficient details reproduce our results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: //ne://///pppppppp cc/
public/ quides, CodeSuumm aasionoolicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht tptt //nips cc/ httpp//bllccccccp waacc//guiieseeeee CCodeeodeuuuaeeeee Subli as onPollo... icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [TODO]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provided details where applicable. In our case, data splits, hyper- parameters, optimizers, etc., are not significant for the implementation the method.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard crror of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report our results with error bars that represent 2 standard deviations from the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http//eerisppe cttp:///lccsssttcccclllleeeeleee
Answer: [Yes]
Justification: We have read the guidelines and we do not think that our work presents any notable concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
29

---

Answer: [Yes]
Justification: We include broader overview statement in the Appendix. We do not expect any negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing asset
Guidelines:
30

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of license ge. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswas aaperswithcode com/ dat satasets has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
The paper does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
31

---

Answer:
[No]
Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "qFlLIVWbJF", "reviewer": "Reviewer_J4GZ", "summary": "This work presents a method for evaluating effectiveness of policies across multiple domains using a new graphical criteria and estimators  by combining data from multiple experimental studies. The authors report error analysis of the proposed estimators that gives provides fast convergence guarantee, and additionally share simulation results as well as empirical verification on real data.", "questions": "- Please provide more empirical analysis on the proposed method ideally on more real-world datasets if possible.\n\n- Please discuss the results on ACTG175 datasets and the different pattern observed in Fig3.c.\n\n- Provide more empirical evidence on real-world data.", "limitations": "Yes.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The targeted problem seem to be of importance, and the proposed method provides theoretical error and convergence analysis, backed by empirical results both based on simulation and real data.", "weaknesses": "The the proposed method seems to have important potentials in real world applications, though the majority of the paper has been dedicated to theoretical analysis and the empirical evaluations are very limited.\nThere is little discussion on how this could impact realworld problems, and there is not a detailed analysis on the empirical results of the results on ACTG175 dataset.\n\nFrom figure 3.c, it can be seen a contradictory message that the proposed DML method does not perform better than OM on all datasets, though there is no discussion on that. \n\nalso authors state that \"simulations on real and synthetic data are provided for illustration purposes only.\" which brings the question to what extend this approach is useful in real world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insights. In the following, we aim to answer each one of your concerns in turn. Please let us know if we can provide more details.\n\n---\n\n> __(1)__ the empirical evaluations are very limited. There is little discussion on how this could impact realworld problems, \n> __(2)__ Please provide more empirical analysis on the proposed method ideally on more real-world datasets if possible.\n> __(3)__ Provide more empirical evidence on real-world data.\n\nWe appreciate this feedback. We provide an additional experiment using the real-world dataset called \"Project STAR\" dataset (Krueger & Whitmore, 2001) [1] in the global response (attached pdf). We will add full details about the experiment in the revised version of the paper.\n\n[1] Krueger, Alan B., and Diane M. Whitmore. \"The effect of attending a small class in the early grades on college‐test taking and middle school test results: Evidence from Project STAR.\" The Economic Journal 111, no. 468 (2001): 1-28.\n\n--- \n\n> there is not a detailed analysis on the empirical results of the results on ACTG175 dataset.\n\nPlease note that the data generation process relating to ACTG 175 is described in the Appendix. More detailed discussions on the results of the synthetic and ACTG 175 experiments will be added.\n\n---\n\n> From figure 3.c, it can be seen a contradictory message that the proposed DML method does not perform better than OM on all datasets, though there is no discussion on that.\n\n> Please discuss the results on ACTG175 datasets and the different pattern observed in Fig3.c.\n\n\nThank you for this comment. This observation is expected. The construction of the DML estimator combines aspects of the OM estimator and the PW estimator. The error of the DML estimator can be represented as a product of the errors of nuisances for the OM estimator and the errors of the PW estimator. As a result, if the PW estimator has a large error, this can be reflected in a correspondingly larger error of the DML estimator. Since the error of the PW estimator is relatively larger than that of the OM estimator in Figure 3c, we expect the DML estimator to underperform compared to the OM estimator when the sample size is small. However, when both the OM and PW estimators converge to the truth, we expect that the DML estimator will outperform the other estimators as the sample size grows.\n\n--- \n\n> also authors state that \"simulations on real and synthetic data are provided for illustration purposes only.\" which brings the question to what extend this approach is useful in real world scenarios. \n\nPlease consider the context in which this statement is located. The full sentence is in our *Broader Impact Statement* section as follows:\n\n_\"Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process.\"_\n\nWith these sentences, we do not question the usefulness of our methods, which we have justified through a series of theoretical and empirical evidence. Rather, we clarify that our paper should not be read as an endorsement of specific policies.\n\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgEAaBDBI7", "reviewer": "Reviewer_cwjE", "summary": "The paper studies off-policy evaluation in a transfer learning setting with multiple source datasets collected from observational and/or randomized studies. The objective is to evaluate the effect of a target policy on a possibly different target population. To achieve this, the author(s) assume at each time point, at least one source population shares the same distribution of the dynamic variable given the historical covariate-treatment pair as the target population. This ensures the identifiability of the target population's policy value, even no source population perfectly matches its entire distribution over time. The author(s) further propose doubly robust estimators, investigate their theoretical properties and finite sample performance.", "questions": "Would it be possible to establish the semiparametric efficiency of the proposed estimators?", "limitations": "The limitations have been discussed in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: To the best of my knowledge, the proposed identification formula and the proposed estimators introduced have not yet appeared in the existing literature.\n\n**Quality**: The theorems established seem to be correct, and the methodologies proposed are theoretically sound and potentially useful.\n\n**Clarity**. Overall, the paper is well-organized and easy to follow.", "weaknesses": "I have two major concerns, regarding the technical assumptions and numerical experiments, along with several moderate comments. I detail them one by one below:\n\n**Assumptions**. The assumptions might appear overly realistic:\n\n* **Same number of studies to horizon**. In addressing a $T$-horizon off-policy evaluation (OPE) problem, the author(s) assume that there are exactly $T$ studies, each corresponding to a time point $t$, where exactly one study per time point shares the same outcome distribution (the distribution of $Y_T$ if $t=T$ and that of $W_{t+1}$ otherwise) with the target population at that time. These settings seem unrealistic in practical scenarios. It would be beneficial for the author(s) to provide examples of real applications that validate this assumption. The current analysis using the ACTG study seems artificial; I will elaborate on this concern in more detail later.\n* **Knowledge of matching source population**. Additionally, the author(s) seem to require prior knowledge of which source population matches the target population’s outcome distribution at each time point. In practical scenarios, while we may have access to multiple studies, it is typically unknown which one aligns with the target's outcome distribution at each time point. A more realistic approach would adaptively learn which studies are similar to the target distribution at each time based on the data. This adaptive learning scenario, in my opinion, would better fit real-world applications.\n* **Identical distributions**. Although the outcome distributions between the source and target populations may be similar at each time, they are not necessarily identical. Even if the distributions are not exactly the same, as long as the differences are minimal, it is reasonable to use source data for transfer learning remains. Recent studies have addressed such distributional shifts using regularization or adaptive weighting: \n    - https://arxiv.org/pdf/2112.09313\n    - https://arxiv.org/pdf/2111.15012\n    - https://arxiv.org/pdf/2406.00317\n\n**Numerical experiments**. The experiments are overly simplified:\n\n* **Single-horizon settings.** While the paper studies multi-horizon dynamic treatment regimes, the simulations are conducted in a single stage setting. I would suggest to use D4RL benchmark datasets or OpenAI Gym environments to evaluate the proposed methodologies in multi-stage studies.\n* **Lack of competing methods.** There are some naive estimator to consider in these transfer learning settings. However, the author(s) did not include them in the experiments. For instance, one can ignore the differences in the outcome distributions and assume the multiple datasets come from a single population, with a mixture of multiple behavior policies that generates the action. \n* **Real data**. The ACTG dataset is generated under a single population. To evaluate the proposed methodology, the author(s) manually created a second population and a target population. It would be better if a real \"multi-source\" dataset could be used.\n\n**Related works**. \n* Under the identical distribution assumption, this paper mainly considers settings with covariate shift. There are some recent works that studied similar problems, although in single-stage settings, e.g.,  https://onlinelibrary.wiley.com/doi/epdf/10.1111/biom.13583. \n* In the related work section, the author(s) argued that their work can be viewed as a bridge between causal inference and OPE by  leveraging formal theories in causal inference to solve OPE problems. In that sense, there are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature uses the backdoor adjustment to learn the Q- or value as a function of the state, to address the confounding effects (see e.g., https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf). Meanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders (https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2110878). Finally, some works have leveraged double negative controls for OPE (https://cdn.aaai.org/ojs/6590/6590-13-9815-1-10-20200520.pdf). \n\n**Typo**: $Z$ is used to denote the dynamic covariate in Section 2. However, this notation has been replaced with $W$ starting from Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your extensive review, we really appreciate the feedback. In the following, we comment on the mentioned weaknesses and address outstanding questions and concerns separately.\n\n---\n\n> __(1)__ Same number of studies to horizon __(2)__ While the paper studies multi-horizon dynamic treatment regimes, the simulations are conducted in a single stage setting.\n\nWe appreciate the careful reading of our setting and thank you for raising this point. Given that the paper tackles a novel problem setting, we have presented a canonical case of the T-horizon off-policy evaluation problem that assumes access to T studies. This assumption can be relaxed in practice, depending on the causal structure. For concreteness, in the attached PDF, we provide a simple example where a two-horizon OPE problem can be solved with access to a single source domain. More generally, the number of heterogeneous domains and the horizon are not necessarily matched if a given causal structure satisfies the proposed criterion formalizing the invariance of mediated and outcome variables across domains.\n\n---\n\n> Knowledge of matching source population.\n\nIn the worst case, where every variable has arbitrarily different distributions from those of the target domain, the data obtained from the source population may not provide any clues for estimating the quantity defined in the target domain. To transfer knowledge, some form of invariance between sources and the target domain is essential. \n\nIn our proposed framework, the required knowledge about domain discrepancy is relatively minimal. Instead of quantitative information, qualitative information on which variables' distributions could differ from the source is sufficient to apply the proposed method. For example, if the age distributions in London and New York are different, we can use this information without needing to know the extent of the difference. \n\n\n\n--- \n\n> A more realistic approach would adaptively learn which studies are similar to the target distribution at each time based on the data.\n\nThank you for the intriguing proposition. In the setting we consider, however, we do not have access to data from the target population (except for baseline covariates) – and therefore, adaptive learning, which requires sources from the target population, may not be directly applicable to our problem setting.\n\nWe want to remark that our problem setting might be reasonable for safety-critical or costly applications where inference of causal effects is necessary before implementing a candidate policy of interest and collecting data in the target domain.\n\n---\n\n> Although the outcome distributions between the source and target populations may be similar at each time, they are not necessarily identical. \n\nWe appreciate the references provided and will cite them. However, we respectfully disagree with the statement. The invariance of outcome distributions across domains stems from the ignorability assumption, which is present in all mentioned papers (e.g., Assumption 1 in the first paper) and in this paper (fourth bullet of Def. 4). This assumption establishes conditional independence between the outcome variable and domain-representing index variables, ensuring consistent outcome distributions across domains.\n\n---\n\n>  D4RL benchmark datasets or OpenAI Gym environments\n\nWe appreciate these suggestions. However, the proposed benchmark may not be suitable for our problem setting, as it lacks elements like heterogeneous population inference, unobserved confounding, or causal structures. \n\nInstead, to further convey the practical utility of the proposed approach, we conduct an additional experiment using a real-world dataset from Project STAR dataset (Krueger & Whitmore, 2001). This includes multiple stages and a semi-synthetic set-up (in which the existing dataset is partitioned to form different environments). We include a full description in the global response and attached pdf.\n\n---\n\n>  For instance, one can ignore the differences in the outcome distributions and assume the multiple datasets come from a single population\n\nWe added the simulation under the suggested scenario, and the result is reported in the attached PDF (Figure 3) in the global response. The key observation in this figure is that the errors do not converge to zero and diverge as the sample size increases. This shows that the estimators are not consistent, unlike the proposed estimators that are consistent with the true effect of the policy in the target domain.\n\n---\n\n> Real data\n\nTo enhance the practical implications of the proposed method, we added a real-world data simulation using the dataset from Project STAR (Krueger & Whitmore, 2001) in the attached pdf (Figure 2). \n\n--- \n\n> Related works.\n\nThanks. We will definitely cite the introduced paper. \n\n--- \n\n> Would it be possible to establish the semiparametric efficiency of the proposed estimators?\n\nYes. Since the proposed estimator is composed of heterogeneous datasets from multiple domains, multiple sampling distributions $P^i$ corresponding to each dataset should be leveraged to construct the semiparametric efficiency bound. Specifically, the partial influence function [2], an influence function defined relative to each $P^i$, can be constructed to provide an efficiency bound for the part corresponding to $\\mathbb{E}_{\\mathcal{D}_i}$. We will add a detailed discussion on the semiparametric efficiency using the partial influence function in the revised version of the paper.\n\n[2] Pires, Ana M., and João A. Branco. \"Partial influence functions.\" Journal of Multivariate Analysis 83.2 (2002): 451-468.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZaowjovvi", "reviewer": "Reviewer_xiN2", "summary": "The paper develops novel graphical criteria and estimators for evaluating the effectiveness of various policies by integrating data from multiple experimental studies.  Through theoretical analysis and empirical validation via simulations, the paper demonstrates the robustness and fast convergence of the proposed estimators.", "questions": "Consider example 3. The proposed framework allows researchers to evaluate a policy from the three experiments. Suppose that all the datasets are synthetic together such that the resulting data is treated as a single dataset obtained from one experiment; probably, we can evaluate a policy based on the synthetic dataset as well. Could you conduct a study that compares the two policy evaluation methods?\n\nIn some situations, it is necessary to evaluate a policy when the size of the multiple treatments is large. Consider example 3 again, but this time, we consider all the big cities in the US instead of three cities in the US. The computation complexity increases, obviously. Are there any efficient methods to handle the relevant problem?\n\nThe adjustment criterion assumes that the distribution is invariant across various experiments. To my realization, the distributions vary even in the examples presented in the paper. How does the proposed methodology handle situations where the underlying assumptions are violated?", "limitations": "The paper focuses on combining two experiments for policy evaluation only. It would be better if the authors can discuss the optimal policy when combining two experiments.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors develop nonparametric identification criteria to determine whether the effect of a policy may be expressed through an adjustment formula from two separate distributions induced by policy interventions collected from different populations. Further, they generalize the identification criteria and propose a general multiplier robust estimator applicable to the evaluation of policies from multiple source datasets.", "weaknesses": "The paper is difficult to follow. It is very hard to understand. Some grammatical mistakes and confused notations, e.g.,\n1. line 71, ‘variable $\\mathbf{Z}$ For example’ -> ‘variable $\\mathbf{Z}$. For example’\n2. unify the notations if necessary, try to avoid writing $\\bigtriangleup_{ij}$ and $\\bigtriangleup_{i, j}$ simultaneously.\nFor the experimental section, please give detailed descriptions on the simulated experiment and the empirical experiment (if the descriptions are too long, please move to the appendix), which allows reader to understand the advantages of the proposed framework.\nFor the theoretical section, the authors first give study when combining two experiments, followed by the study when combining multiple experiments. Honestly, the materials in the two studies are similar. Authors can unify the two studies: presenting the study when combining multiple experiments followed by the two examples (example 2 and example 3). This can release more space to discuss the core components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. Please find below a point by point answer to all questions and concerns. Please let us know if we can clarify anything further.\n\n---\n\n> __(1)__ line 71, ‘variable $\\mathbf{Z}$ For example’ -> ‘variable $\\mathbf{Z}$. For example’; __(2)__ unify the notations if necessary, try to avoid writing $\\Delta_{ij}$ and $\\Delta_{i,j}$ simultaneously.\n\nThank you for pointing this out. We will fix these. \n\n---\n\n> For the experimental section, please give detailed descriptions on the simulated experiment and the empirical experiment \n\nThank you for the feedback. We will add further description on the details of the simulations. For further reference, we have provided the details of the dataset leveraged in this paper in the appendix. \n\n---\n\n> Honestly, the materials in the two studies are similar. Authors can unify the two studies: presenting the study when combining multiple experiments followed by the two examples (example 2 and example 3). This can release more space to discuss the core components.\n\nThank you for your suggestion. We designed the current structure of the paper since our theories are new to both the causal and RL communities. We wanted to introduce the idea starting from a canonical setting (combining two experiments) and then extending it to more general settings.\n\n\n---\n\n> Suppose that all the datasets are synthetic together such that the resulting data is treated as a single dataset obtained from one experiment\n\nWe added the simulation under the suggested scenario, and the result is reported in the attached PDF (Figure 3) in the global response. The key observation in this figure is that the errors do not converge to zero and diverge as the sample size increases. This shows that the estimators are not consistent, unlike the proposed estimators that are consistent with the true effect of the policy in the target domain.\n\n\n\n---\n\n> The computation complexity increases\n\nThe computational complexity of the proposed estimator is $O(m \\cdot T(n,m))$, where $m$ is the number of variables in a causal graph, $n$ is the number of samples, and $T(n,m)$ is the time complexity of training ML models for estimating nuisances, which is commonly polynomial in $n$ and $m$. The complexity is $O(m \\cdot T(n,m))$ because we are learning $2 \\times L \\times m$ nuisances for constructing the proposed estimator. This analysis shows that even if the number of source domains (upper bounded by $m$) increases, the time complexity increases linearly with $m$. Therefore, the proposed estimator is scalable with respect to the number of source domains.\n\n---\n\n> To my realization, the distributions vary even in the examples presented in the paper. \n\nWe couldn't parse this question. In our setting, the distributions vary across domains. However, if some invariances between domains (such as the domain transfer for $Y$ and $W$ in Def. 4 and Def. 6) hold, our proposed method is applicable.\n\n---\n\n> How does the proposed methodology handle situations where the underlying assumptions are violated? \n\nIf the assumptions are violated, the proposed estimator will not be consistent with the effect of the policy in the target domain and may deviate systematically from the true value. For example, as shown in the attached PDF in the global response, if we ignore the heterogeneity assumption and treat all datasets as if they are from the same population, the resultant estimates do not converge and even diverge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oc5oq4Txd6", "reviewer": "Reviewer_GRsa", "summary": "This paper studies how to evaluate policies where source and target sites have distribution shifts. The authors introduce identification criteria for the effectiveness of policies, and develop doubly robust estimators that achieves fast convergence. The results are also generalized to multiple source datasets. Simulation results are provided to show the effectiveness of the method.", "questions": "See weaknesses part.", "limitations": "N/A", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and easy to follow. The setups are clearly introduced and motivated, the assumptions and methodologies are accurately stated. \n\n2. The proposed framework is general to cover both two domains and multiple domains.\n\n3. The empirical results seem good and well aligned with the theories for both synthetic simulation and real world datasets.", "weaknesses": "I am not an expert in this field and not familiar with causal inference. I find no major weaknesses. Some minor weaknesses are on the empirical evaluations. For example, the non-synthetic experiments are only conducted on ACTG 175 clinical trial dataset. Experimenting on other different datasets will enhance the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We agree with your comment and have added another experiment using a real-world dataset.\n\n--- \n\n> For example, the non-synthetic experiments are only conducted on ACTG 175 clinical trial dataset. Experimenting on other different datasets will enhance the empirical results.\n\nWe appreciate this feedback. We provide an additional experiment using the real-world dataset called \"Project STAR\" dataset (Krueger & Whitmore, 2001) [1] in the global response (attached pdf). We will add full details about the experiment in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you again for your time and dedication in reviewing our work. In this global response, we describe additional results, illustrated with figures in the attached PDF, to address outstanding comments and questions. In particular, we attach three figures that are described below.

---

**Figure 1: Relaxation of the Assumption on the number of source datasets**

To respond the Reviewer cwjE's concern on "_the author(s) assume that there are exactly $T$ studies_", we describe a scenario in which a single source domain can be used to evaluate a two-stage policy. This shows that the assumption of having exactly $T$ source datasets for solving a $T$-stage Offline Policy Evaluation problem can be relaxed in practice, depending on the causal structure of the domains.

---

**Figure 2: Experiment on Project STAR dataset**

The second figure describes an additional experiment evaluating the effect of policies determining teacher/student ratios to improve academic achievement. Specifically, we use a semi-synthetic version of the Project STAR dataset (Stock et al., 2007) [1]. This study investigated the impact of teacher/student ratios on academic achievement for kindergarten through third-grade students. It was a four-year longitudinal study where students were randomly assigned to one of three interventions with different class sizes each year, following different randomization procedures. The dataset is publicly accessible from the R data repository.

The causal diagram we assume for this setting is provided in Fig. 2 of the attached PDF. Specifically, we consider the evaluation of a 3-stage policy that determines student-teacher ratio across three different grades over time (these are the variables $(X_1, X_2, X_3)$ in the target environment). We observe intermediate outcomes $(W_1, W_2)$ that represent academic scores in grades 0 (Kindergarten) and 1, baseline covariates $\mathbf{C}$ such as ethnicity and gender, and the outcome of interest $Y$ representing academic scores at the end of grade 3.

To mimic the setting where data at different stages was collected from different domains, we subsample the dataset using different sets of probabilities to induce differences in the distributions of baseline covariates (similar to the procedure conducted for ACTG 175).

We consider a similar evaluation setup as in the main body of this paper and evaluate the proposed estimators PW, OM, and DML on different dataset sizes, plotting their absolute errors compared to the ground truth effect of the candidate policy. The results are shown in Fig. 2 of the attached PDF. We observe that the DML estimator converges faster. The performance pattern is qualitatively similar to the other presented experiments, with all methods improving with sample size towards convergence (in the limit), and the DML estimator converging faster.

[1] Stock, J.H. and Watson, M.W. (2007). Introduction to Econometrics, 2nd ed. Boston: Addison Wesley.


---

**Figure 3. Performance of naive estimators that ignore the differences across domains**

To respond to comments from reviewers cwjE and xiN2, we implemented the "naive" estimator that ignores discrepancies across domains. We considered the 2-stage synthetic simulation scenario described in Sec. 5.1 and plotted performance as a function of increasing sample size. The key observation in this figure is that the errors do not converge to zero and diverge as the sample size increases. This shows that the estimators are not consistent, unlike the proposed estimators that are consistent with the true effect of the policy in the target domain.

Meta-review: All the reviewers agree the problem tackled in the paper, offline policy evaluation in transfer learning setup, is important. There were concerns about the lack of real data evaluation, the assumptions / settings of the theory and missed related works. In addition the paper seems to suffer from presentation.

Most of the concerns have been addressed by the authors’ rebuttal and the reviewers are leaning towards accepting this paper.

I would still urge the authors to improve the presentation and make the paper widely accessible to neurips community.
Decision: Accept (poster)