Efficient Policy Evaluation Across Multiple Different Experimental Datasets

Authors: yonghan  jung
alexis  bellot
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:19:16

Abstract: Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from the same distribution. This assumption is often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability of the effectiveness of policies in a target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies (e.g., conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.

Main paper: 
 introduction: 
 In the empirical sciences, conclusions on the effect of actions or policies is often supported by evidence drawn from prior observations and experiments. The conditions under which such inferences can be formally justified can be traced back (in part) to Campbell, Stanley and Cook [10, 11, 14] . They argued for a basic dichotomy in the kinds of questions that scientists seek to answer from experimental data. On the one hand asking whether "in fact, the experimental stimulus made some significant difference in this specific instance?", and on the other hand asking "to what populations, settings, and treatments can this effect be generalized?" [10, p. 297 ]. These inferences have since been labelled as internal validity and external validity, respectively.External validity is concerned with the extent to which findings from one population can be "reprocessed", or "re-calibrated" so as to circumvent population differences and produce valid generalizations in a target population where experiments cannot be performed (e.g., outside the laboratory, different domains, etc.). The validity of these inferences will necessarily be contingent on a careful analysis to ascertain the commonalities and differences between domains as, for example, if the target domain is completely arbitrary generalization is impossible. In the causal transportability literature, the basis for generalization (also called transportability) is justified by the stability and invariance of the causal mechanisms shared across populations and domains [20, 32] . Several graphical characterizations exist to delineate the conditions under which transportability is possible, with recent algorithms proposing solutions for general instances of the external validity task combining observational and experimental distributions under partial observability [36, 2, 3, 16, 30] .˚Equal contribution. Correspondence to Yonghan Jung: jung222@purdue.edu. : Now at Google DeepMind. Work done partly while affiliated with Columbia University and partly in an independent capacity. These algorithmic solutions express a target policy effect in terms of the observational and experimental source distributions. Still, then one needs to go further and estimate the resulting expression from finite samples. In practice, with a finite number of samples and potentially high-dimensional covariates, estimating causal expressions is quite challenging. Effective estimators have been developed for specific settings, starting with doubly-robust estimators for functionals given by the backdoor criterion [13, 37, 9, 45] , and recently extended to cover general identification scenarios with observational and experimental samples [25, 26, 8] . These techniques also find parallels across other related disciplines, such as reinforcement learning where re-weighting [42, 31] , outcome modelling [6] , and doubly-robust estimation [18] , are common for evaluating the effect of policies to overcome shifts in the behaviour policy. Recently, [44] and [22] have considered policy evaluation under covariate shift and selection bias, a special case of the external validity problem with a given graph. Despite their generality, existing estimators still only cover a limited portion of realistic scientific inferences. In particular, existing methods are not applicable in settings where datasets are collected in different domains.We consider the generalization of causal claims from observational and experimental data through the task of policy evaluation. The target for inference is E P 0 π rY s where P 0 π symbolizes the distribution of data in a target domain (indexed as 0) in which a hypothetical policy of interest π (also known as dynamic treatment regimes [33] or soft interventions [16] ) has been implemented. The question then becomes how to identify and estimate E P 0 π rY s, given finite samples from multiple observational and experimental data (e.g., P i πi , a source domain indexed by i in which experimental policy is π i that may differ with π 0 ) collected under different settings and structural assumptions, encoded in causal diagrams. We aim to bridge the gap between identification and estimation to solve general instances of external validity. Our contributions are twofold:1. Sec. 3: We develop nonparametric identification criteria (Thm. 1) to determine whether the effect of a policy may be expressed through an adjustment formula from two separate distributions induced by policy interventions, collected from different populations. Based on this formulation, we develop a multiply robust estimator (Thm. 3) that enjoys multiply robustness against model misspecification and bias.We use bold letters (X) to denote a random vector and X a random value. Each random vector is represented with a capital letter (X) and its realized value with a small letter (x). Given a set X " tX 1 , ¨¨¨, X n u, we denote X piq :" tX 1 , ¨¨¨, X i u. For a discrete vector X, we use 1 x pXq to represent the indicator function such that 1 x pXq " 1 if X " x; 1 x pXq " 0 otherwise.For comprehensibility, we use P pvq to denote a probability at V at v for discrete/continuous random variables V. In similar, we use ř z for Z Ď V for the summation/integration over a mixture of discrete/continuous random variables Z For example, we write the back-door adjustment as ř z E P rY | x, zsP pzq even when Z is a mixture of discrete/continuous variables. We use E P rf pVqs :" ř v f pvqP pvq for a function f . For a sample set D :" tV piq : i " 1, ¨¨¨, nu where V piq denotes the ith samples, we use E D rf pVqs :" p1{nq ř n i"1 f pV piq q. We use }f } P :" a E P rtf pVqu 2 s. If a function p f is a consistent estimator of f having a rate r n , we use p f ´f " o P pr n q. We say f is L 2 -consistent if } f ´f } P " o P p1q. We use p f ´f " O P p1q if p f ´f is bounded in probability, and p f ´f " O P pr n q when p f ´f is bounded in probability at rate r n .We use Structural Causal Models (SCMs) as our framework [35] . An SCM M is a quadruple M " xU, V, P pUq, Fy. U is a set of latent variables following a joint distribution P pUq. V is a set of observable variables whose values are determined by functions F " tf Vi : V i P Vu such that V i Ð f Vi ppa Vi , u Vi q where PA i Ď V and U Vi Ď U. Each SCM M induces a distribution P pVq and a causal graph G in which directed edges from every variable in PA i to V i exist. Dashedbidirected arrows encode correlated latent variables. 
sec. 4:: 
 We generalize these identification criteria (Thm. 4) and propose a general multiply-robust estimator (Thm. 6) applicable for the evaluation of policies from multiple source datasets.We investigate the sequential decision-making setting concerning a set of actions X, a series of dynamic covariates Z, a series of static covariates C and an outcome variable of interest Y in an SCM M. A policy vector π :" tπ i u over actions X " tX 1 , ¨¨¨, X m u is an ordered set of decision rules for each X i P X. Actions are selected according to a topological ordering X 1 ă ¨¨¨ă X K over time. Each action X i is potentially associated with a set of prior static and dynamic covariates, for example, the decision rule for X k could be defined as x k " πp¨| z pkq , x pk´1q , c pkq q. Every πpX k | Z pkq , X pk´1q , C pkq q is a probability distribution mapping from domains of the set of inputs tZ pkq , X pk´1q , C pkq u to the domain of actions X k . The implementation of a policy π in M induces an intervened model M π , that sets values of every X P X to be decided by the policy π, replacing the functions tf X , X P Xu that would normally set its value. We denote a distribution induced by M π as P π . Now, we fix the notion of the policy evaluation as follows:Definition 1 (Policy evaluation [41] ). The policy evaluation is to predict the effectiveness of a policy vector π on an outcome Y in an target SCM M 0 ; i.e., ψ 0 :" E P 0 π rY s.Difficulties in estimating E P 0 π rY s comes from that the distribution or samples from P 0 π are generally not available. These discrepancies can be formalized under the rubric of SCMs as follows. In the most general setting, an investigator might leverage multiple source domains tM 1 , M 2 , . . . , M K u over V that entail distributions P : tP 1 , P 2 , . . . , P K u. Data or samples from these distributions may be available under different behaviour policies, e.g., π 1 , π 2 , . . . , π K , depending on the study or data collection protocol implemented in each domain (that might include an observational regime, i.e. no policy implemented). To ground the policy evaluation problem, we define graphical tools to capture commonalities and discrepancies across domains.Definition 2 (Domain discrepancy [29] ). For every pair of SCMs M i , M j (i, j P t0, 1, 2, . . . , Ku) defined over V , the domain discrepancy set ∆ ij Ď V is defined such that for every V P ∆ ij there might exist a discrepancy betweenf M i V ‰ f M j V , or P M i pu V q ‰ P M j pu V q.Definition 3 (Selection diagram [29] ). The selection diagram G ∆ " tG j u jPt0,1,2,...,T u Ť tG ∆0j u jPt1,2,...,T u is a graph constructed from G i (i P t0, 1, 2, . . . , T u) by adding the selection node S ij to the vertex set, and adding the edge S ij Ñ V for every V P ∆ ij .∆ i,j locates the mechanisms where structural discrepancies between two domains are suspected to take place. V R ∆ i,j represents the assumption that the mechanisms for V are invariant across the two domains. The induced selection diagram is a parsimonious representation of these constraints.The following example illustrates these notions.Example 1 (External validity under covariate shift). A common instance of the external validity problem in the literature considers the evaluation the effect of a policy π : Ω C ˆΩX Ñ r0, 1s for assigning a treatment X P t0, 1u, subject to shift in the distribution of covariates C. For this example, let source and target domains M : tM 1 , M 0 u over V " tC, X, Y u, U " tU XY , U C u be defined as follows,M 1 : $ ' ' & ' ' % F " $ & % C Ð f C pU C q X Ð f X pC, U XY q Y Ð f Y pX, C, U XY q P pU q " P pU XY qP pU C q M 0 : $ ' ' & ' ' % F 0 " $ & % C Ð f 0 C pU C q X Ð f X pC, U XY q Y Ð f Y pX, C, U XY q P 0 pU q " P pU XY qP 0 pU C qHere, C P ∆ 1,0 , tY, Cu R ∆ 1,0 as only the mechanism for C varies across domains. Consider the evaluation of π : πpX " 1 | cq :" 1{p1 `expt´cuq given an experimental dataset in M 1 in which X has been randomized, i.e., X " Bernp0.5q, and covariate data P 0 pCq available in M 0 . Notice that we do not have access to the specification of the SCMs M, but only the induced diagrams G ∆ , and a subset of entailed distributions P : tP 1 randpXq pX, Y, Cq, P 0 pCqu. The policy effect is expressible asE P 0 π rY s " ÿ x,c,y yP 1 randpxq py | c, xqπpx | cqP 0 pcq,and estimated given the policy π and the combination of the available data from P 1 , P 0 . ■ X 1 W X 2 C 1 C 2 Y π 1 (a) G 1 π 1 X 1 W X 2 C 1 C 2 Y π 2 (b) G 2 π 2 X 1 W X 2 C 1 C 2 Y S C1 S C2 S Y π 0 π 0 (c) G ∆ 0,1 π 0 X 1 W X 2 C 1 C 2 Y S C1 S C2 S Y π 0 π 0 (d) G ∆ 0,2 π 0W : pW K K S | C, X 1 q in G ∆0,1π0 ; i.e., the distribution over W is invariant between the source distribution from M 1 and the target.i for i " 1, ¨¨¨, m ´1: pW i K K S | C piq , W pi´1q q in G ∆0,iπ0 ; i.e., the distribution over W i is invariant between the source distribution from M i and the target.3. Adjustment for Y : pY K K π i | C, W, Xq in G πi for i P t0, mu; i.e., the distribution over Y is invariant between regimes π 0 and π m .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We include a broader overview statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability in real-world settings.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
combining experiments from two domains: 
 Example 1 illustrates two challenges in combining data from different domains to infer the effect of a new policy in a target domain. In a first instance highlighting the challenge of identification, that is inferring an expression in terms of P that identifies the policy effect, and in a second instance highlighting the challenge of estimation, that is providing efficient estimators from finite samples for the identified policy effect. The following example will serve to motivate this setting. Example 2 (Two-stage treatment strategies). A team of physicians is contemplating a treatment plan π 0 against heart disease Y for their patients in M 0 . They consider administrating two drugs in sequence: a drug against hypertension X 1 , followed by an anti-diabetic drug X 2 depending on the effect of X 1 on blood pressure W . To support their evaluation, two studies exist on these drugs, from domains M 1 , M 2 , that, however, have only analyzed their effect in isolation (on X 1 and X 2 separately) and under different treatment guidelines, π 1 , π 2 respectively. The data collected refers to the variables V :" pY, C 1 , C 2 , X 1 , X 2 , W q in which pC 1 , C 2 q are demographic variables. Formally, we assume physicians have access to P : tP 1 π1 pVq, P 2 π2 pVq, P 0 pC 1 , C 2 qu. The superscripts in P 0 , P 1 , P 2 are the index for the domain, and the subscripts π 0 , π 1 , π 2 denote the policies for assigning treatments. G ∆ in Fig. 1 encodes the structural assumptions, which include discrepancies across domains and implemented policies in the available data. For example, the graph G 1 π1 specifies the known guideline π 1 used in M 1 , while no specific plan was followed for the assignment of X 2 , that in practice depends on the patient's covariates C 2 as well as unobserved factors, e.g. mood, health awareness, etc. (summarized in the bi-directed arc). In addition, selection diagrams describe differences between domains. For example, the edge tSC1 Ñ C 1 u in G ∆0,1 π0indicates a potential change in the distribution of covariates C 1 across domains M 0 , M 1 . The question then becomes how to estimate E P 0 π 0 rY s given pG ∆ , Pq. ■Example 2 illustrates the complexity of drawing inferences from multiple datasets collected under different settings. We extend this example to provide a general identification procedure for the effect of policies when two source datasets subject to different policies and/or discrepancies with the target domain are available. Let V :" pY, C, X 1 , W, X 2 , W, Y, Sq denote a set of disjoint variables, where Y is an outcome variable, C, pC, Wq are covariates corresponding to two experiments, pX 1 , X 2 q are treatment variables, and S denotes the selection nodes describing discrepancies across pairs of domains. Formally, the task signature is given as follows:• Input: Samples from P " tP 1 π1 pVq, P 2 π2 pVq, P 0 pC 1 , C 2 qu; structural assumptions G ∆ :"tG 0 π0 , G 1 π1 , G 2 π2 , G ∆0,1 π0 , G ∆0,2π0 u. • Query: Estimate E P 0 π 0 rY s where P 0 is distribution on the target domain and π 0 is a target policy assigning treatments with π 0 pX 1 | C 1 q and π 0 pX 2 | C 2 , W q.Given these inputs, a sufficient condition for identifying the query is given as follows: Definition 4 (Adjustment criterion for combining two experiments). Given G ∆ , the adjustment criterion for combining two experimental datasets is defined by the following d-separation statements:1. Domain transfer for Y : pY K K S | C, X 1 , X 2 , W q in G ∆0,2π0 ; i.e., the distribution over Y is invariant between the source distribution from M 2 and the target.Y : pY K K π i | C 1 , C 2 , X 1 , X 2 ,W q in G πi for i P t0, 2u; i.e., the distribution over Y is invariant between regimes π 0 and π 2 .4. Adjustment for W : pW K K π i | C 1 , C 2 , X 1 q in G πi for i P t0, 1u; i.e., the distribution over W is invariant between regimes π 0 and π 1 .The adjustment criterion could be shown to hold for Example 2. Specifically, domain transfer Y could be shown by inspecting Fig. 5e as the set S " tS C1 , S C2 , S W u is d-separated from Y , conditional on tC 1 , C 2 , X 1 , X 2 , W u. Similarly, domain transfer for W holds as S " tS C1 , S C2 , S Y u is d-separated from W , given C 1 , C 2 , X 1 in Fig. 5d . Similarly, one could verify the adjustment condition for Y by inspecting Fig. 5c and the adjustment condition for W by inspecting Fig. 5a .For this example, these conditions imply identifiability of the target query E P 0 π 0 rY s given pG ∆ , Pq. Theorem 1 (Adjustment for combining two experiments). Under the adjustment criterion in Def. 4, the target query ψ 0 :" E P 0 π 0 rY s is identifiable from the samples from P 1 π1 pVq, P 2 π2 pVq, P 0 pC 1 , C 2 q. Specifically, it's expressed as follows 3 :EQUATIONwhere X :" pX 1 , X 2 q and C :" pC 1 , C 2 q.Effectively, despite the differences across domains encoded in Example 2, the effect of the new combination of anti-diabetic and anti-hypertensive drugs π 0 , can be estimated using samples from experiments already conducted in M 1 , M 2 , and baseline characteristics of patients in M 0 .This section considers the estimation of the effect of policies, building on the identification criterion in Thm. 1. We first parameterize the identification estimand in Eq. ( 1) with two types of nuisance parameters µ and ω. µ is a collection of regression parameters, and ω is a collection of the ratio of distributions.The regression nuisance parameters are defined as follows: µ 2 0 pC, W, Xq :"E P 2 π 2rY | C, W, Xs and μ2 0 pC, W, X 1 q :" ř x2 µ 2 0 pC, W, X 1 , x 2 qπ 0 px 2 | C, W, X 1 q. Recursively, µ 1 0 pC, X 1 q :" E P 1 π 1 rμ 2 0 pC, W, X 1 q | C, X 1 s and μ1 0 pCq :"ř x1 µ 1 0 pC, x 1 qπ 0 px 1 | Cq.Eq. ( 1) can be parameterized as E P 0 π 0 rY s " E P 0 rμ 1 0 pCqs.On the other hand, the ratio nuisance parameters ω 2 0 , ω 1 0 are defined as functionals satisfying the following properties:EQUATIONA closed form of the ω i 0 is provided in the later section at Eq. (14) . By the definition of ratio nuisances, Eq. ( 1) can be parameterized asE P 0 π 0 rY s " E P 2 π 2 rω 2 0 pC, W, XqY s.Equipped with these nuisances, we now present the DML-based estimator for the target query: Definition 5 (DML for combining two experiments). Let D 2 " P 2 π2 pVq, D 1 " P 1 π1 pVq and D 0 " P 0 pCq. Let L ě 2 denote a fixed number.1. Sample split: For ℓ " 1, ¨¨¨, L, randomly split D i for i P t0, 1, 2u into L-fold. The ℓ'th partition of the sample is denoted D i ℓ . The complement is D i ´ℓ :" D i zD i ℓ .2. Nuisance estimation: For each ℓ " 1, ¨¨¨, L, learn the estimator model μ2 ℓ and μ1 ℓ for µ 2 0 , µ 1 0 using samples D 2 ´ℓ, D 1 ´ℓ, respectively. Also, learn the estimation model for ω1 ℓ , ω2 ℓ for ω 1 0 , ω 2 0 using samples D i ´ℓ for i " 0, 1, 2, respectively.The DML estimator ψ for E P 0 π 0 rY s is then given asEQUATIONEstimating the ratio nuisance tω 1 , ω2 u can be challenging due to the necessity of estimating density ratios likeP 1 π 1 pCq P 2 π 2 pCq or P 1 π 1 pW |C,X1q P 2 π 2pW |C,X1q . We employ the classification-based method for estimating the density [17, Sec. 5.4 ]. To illustrate this method, consider estimatingP 1 π 1 pCq P 2 π 2pCq . We assign λ " 1 if samples of C are from P 1 π1 and λ " 0 if from P 2 π2 . Then, it's provable thatP 1 π 1 pCq P 2 π 2pCq " P pλ"1|Cq P pλ"0|Cq , which can be estimated using off-the-shelf probabilistic classification estimators.The error of the DML estimator is presented below:Theorem 2 (Learning Guarantees). Suppose μ2 ℓ , μ1 ℓ ă 8 and 0 ă ω2 ℓ , ω1 ℓ ă 8. Define ϕ 2 pV; µ 2 , π 2 q :" ω 2 pC, W, XqtY ´µ2 pC, W, Xqu, ϕ 1 ppC, W, X 1 q; μ2 , µ 1 , ω 1 q :" ω 1 pC, X 1 qtμ 2 pC, W, X 1 q ´µ1 pC, X 1 qu, and ϕ 0 pC; μ1 q :" μ1 pCq ´ψ0 . For i " 0, 1, 2, define ϕ i 0 as ϕ i equipped with true nuisances (µ i 0 , π i 0 ) and φi ℓ as ϕ i equipped with estimated nuisances μi ℓ , πi ℓ . Define R i :" p1{Lqř L ℓ"1 pE D i ℓ r φi ℓ s ´EP i r φi ℓ sq for i " 0, 1, 2. Then, 1.The error ψ ´ψ0 is decomposed as follows:EQUATION2. Let ρ 2 i,0 :" V P i π i rϕ i 0 s. With probability (W.P) greater than 1 ´ϵ,2 ÿ i"0 R i ď 3 c 2 ϵ ¨g f f e 2 ÿ i"0 ρ i,0 |D i | `g f f e L ÿ ℓ"1 2 ÿ i"0 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ | ‹ ‚.(5)3. Let κ 3 i,0 :" E P i π i r|ϕ i 0 | 3 s.Let Φpxq denote the standard normal CDF. W.P greater than 1 ´ϵ,EQUATIONIf the nuisance parameters μi ℓ and πi ℓ converge at a rate of n ´1{4 (where n is the size of the smallest sample set), the DML estimator achieves a faster convergence rate of n ´1{2 . This rapid convergence allows its asymptotic distribution to closely approximate the standard normal distribution, as is further clarified in the asymptotic analysis: Theorem 3 (Asymptotic Error). Suppose each nuisance estimates μ2 ℓ , μ1 ℓ , ω2 ℓ , ω1 ℓ are L 2 -consistent and bounded. Then, the error of the DML estimator ψ in Def. 5 is given as follows:EQUATIONwhere R i converges in distribution to normalp0, ρ 2 i,0 q.Eq. ( 7) implies that the error term ψ ´ψ0 converges to zero faster than the convergence rate of nuisances, which is a property known as debiasedness. C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 1 (a) G 1 π 1 C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 2 (b) G 2 π 2 C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 3 (c) G 3 π 3 C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 0 π 0 π 0 S C1 S C2 S C3 S W2 S Y (d) G ∆ 0,1 π 0 C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 0 π 0 π 0 S C1 S C2 S C3 S W1 S Y (e) G ∆ 0,2 π 0 C 1 X 1 W 1 C 2 X 2 W 2 C 3 X 3 Y π 0 π 0 π 0 S C1 S C2 S C3 S W1 S W2 (f) G ∆ 0,3 π 0The DML estimator ψ for E P 0 π 0 rY s is then given asEQUATION)The error of the DML estimator is presented below:Theorem 5 (Learning Guarantees). Suppose µ i 0 , μi ℓ ă 8 and 0 ă π i 0 , πi ℓ ă 8 almost surely for i " 1, ¨¨¨, m. Define ϕ i ppC, W piq , X piq q; ω i , μi`1 , µ i q :" ω i tμ i`1 ´µi u for i " 1, ¨¨¨, m, where μm`1 :" Y . Let ϕ 0 pC; μ1 q :" μ1 ´ψ0 . For i " 0, ¨¨¨, m, define ϕ i 0 as ϕ i equipped with true nuisances, and φi ℓ as ϕ i equipped with estimated nuisances. Define R i :" p1{Lqř L ℓ"1 pE D i ℓ r φi ℓ s ÉP i r φi ℓ sq for i " 0, 1, ¨¨¨, m. Then, 1.The error ψ ´ψ0 is decomposed as follows:EQUATION)2. Let ρ 2 i,0 :"EQUATION)3. Let κ 3 i,0 :" E P i π i r|ϕ i 0 | 3 s.Let Φpxq denote the standard normal CDF. W.P greater than 1 ´ϵ,EQUATION)A corresponding asymptotic error analysis is following: Theorem 6 (Asymptotic Error). Suppose each nuisance estimates μ1 ℓ , ¨¨¨, μm ℓ and ω1 ℓ , ¨¨¨, ωm ℓ are L 2 -consistent and bounded. Then, the error of the DML estimator ψ in Def. 7 is given as follows:EQUATION)where R i converges in distribution to Normalp0, ρ 2 i,0 q.Similarly to Thm. 3, this result implies that the DML estimator ψ converges fast even when the nuisance estimates converge relatively slowly.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All theoretical statement are quoted in full in the paper. We have attempted to provide an example to illustrate the significance of each theoretical statement and highlight its implications. The formal proof of all statements is given in the Appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
combining multiple experiments: 
 In this section, we extend our method to incorporate the combination of data from multiple experiments, specifically focusing on m different experiments derived from varied policies (π i ) in distinct source domains (M i ). A practical scenario for this task is the following: Example 3 (Multi-stage treatment strategies). Consider a scenario involving hospitals in three different cities: New York (domain 1 with P 1 , G 1 ), Los Angeles (domain 2 with P 2 , G 2 ), and San Francisco (domain 3 with P 3 , G 3 ). Each hospital has different guidelines, i.e., policies, for diabetes treatment. In New York, the hospital focuses on insulin therapy adjustment based on the patient lifestyle choices, primarily for Type 1 Diabetes patients (π 1 ). In Los Angeles, the hospital focuses on team diet and exercise regimen adjustments, primarily for Type 2 Diabetes patients (π 2 ). In contrast, San Francisco's approach involves advanced monitoring and AI-driven predictive adjustments for higher-risk diabetes patients. Now, as the leader of a new clinical team in Chicago (the target domain with P 0 , G 0 ), the task is to evaluate a novel candidate treatment policy π 0 , which integrates the strategies from these three domains to provide comprehensive care for both Type 1 and Type 2 Diabetes patients. The structure of the problem is captured in causal diagrams in Fig. 2 , illustrating the data-generating process, the experiments in each city, and the assumed discrepancies between these source domains and Chicago. ■We consider a sequence of variables pC, X 1 , W 1 , ¨¨¨, X m , W m :" Y q where pC, W pi´1q q represent the covariates corresponding to each of the i'th experiments, and pX 1 , ¨¨¨, X m q are the corresponding treatment variables. We are given samples drawn from P i πi pVq for i " 1, ¨¨¨, m and P 0 pCq. We will leverage causal diagrams G πi and selection diagrams G ∆0,i for every i " 1, ¨¨¨, m. Formally, the task signature is given as follows:• Input: Samples from P i πi pVq for i " 1, ¨¨¨, m andP 0 pC 1 , C 2 q; Causal diagrams G i πi and selection diagrams G ∆0,i π0 for i " 1, ¨¨¨, m.• Query: Estimate the effect of the target policy π 0 on the target domain M 0 ; i.e., E P 0 π 0 rY s. ; i.e., the distribution over Y is invariant between the source distribution from M m and the target.W i i " 1, ¨¨¨, m ´1: pW i K K π j | C piq , X pi´1qq in G πj for j P t0, iu; i.e., the distribution over W i is invariant between regimes π 0 and π i .These conditions lead to the following identification criterion. Theorem 4 (Adjustment for combining multiple experiments). Under the adjustment criterion in Def. 6, the target query ψ 0 :" E P 0 π 0 rY s is identifiable from the samples from P 1 π1 pVq, ¨¨¨, P m πm pVq and P 0 pCq. Specifically, it's expressed as follows:EQUATIONThe regression nuisance parameters are defined as follows. We first define the following nuisance.EQUATIONEQUATIONFor i " m ´1, ¨¨¨, 1, the other nuisances are defined in a following manner:EQUATIONEQUATION)We note that Eq. ( 8) can be parameterized as E P 0 π 0 rY s " E P 0 rμ 1 0 pCqs. On the other hand, the ratio nuisance parameters ω i 0 for i " 1, ¨¨¨, m are defined as functionals satisfying the followings:EQUATIONwhere the closed form is given asEQUATIONEq. ( 8) can be parameterized as E P m πm rω pmq pC, W pm´1q , X pmq qY s. Equipped with these nuisances, we define a corresponding estimator as follows. Definition 7 (DML for combining multiple experiments). Let D i " P i πi pVq for i " 1, ¨¨¨, m and D 0 " P 0 pCq. Let L ě 2 denote a fixed number.1. Sample split: For ℓ " 1, ¨¨¨, L, randomly split D i for i P t0, 1, ¨¨¨, mu into L-fold. The ℓ'th partition of the sample is denotedD i ℓ . The complement is D i ´ℓ :" D i zD i ℓ .2. Nuisance estimation: For each ℓ " 1, ¨¨¨, L, learn the estimator model μm ℓ , ¨¨¨, μ1 ℓ for µ m 0 , ¨¨¨, µ 1 0 using samples D m ´ℓ, D 1 ´ℓ, respectively. Also, learn the estimation model for ω1 ℓ , ¨¨¨, ωm ℓ for ω 1 0 , ¨¨¨, ω m 0 using samples D i ´ℓ for i " 0, 1, ¨¨¨, m, respectively.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide the data generating mechanisms, details of the target of estimation and information as to what python libraries can be used to fit the proposed estimators. We do not, however, disclose an open source implementation of the proposed methods at this moment. 
experiments: 
 In this section, we demonstrate the proposed estimators in Defs. (5, 7) for combining multiple experimental datasets from different domains. We first compared the estimators on synthetic data to provide evidence of the fast convergence and doubly robustness behaviours of the proposed estimators. We conclude with an analysis of the ACTG 175 clinical trial [21] and Project STAR. We will use T est pxq for est P treg, pw, dmlu to denote the estimators tOM, PW, DMLu for the policy effect E P 0 π 0 Y . OM and PW estimators are purely based on the regression-based nuisances µ and ω, respectively. To assess the quality of each estimator, we consider the absolute error (AE) as AE est " |T est pxq ´EP 0 π 0 rY s |. We used XGBoost [12] to estimate nuisances.We ran 100 simulations for each N " t2500, 5000, 10000, 20000u where N is the sample size. We measure the AE est in the presence of the 'converging noise ϵ' in estimating the nuisance, decaying at a N ´1{4 rate (i.e., ϵ " normalpN ´1{4 , N ´1{4 q, where N is the size of samples). To enforce the convergence rate of nuisance estimates no faster than the decaying rate n ´1{4 , we add ϵ to all nuisance estimates. This scenario is inspired by the experimental design discussed in [27] . The AE plots for combining two/multiple experiments are presented in Figs. (3a, 3b ). For all examples, the proposed DML estimator outperforms the other two estimators by achieving fast convergence. This result corroborates the robustness property in Thm. (3, 6) , which implies that the proposed estimator converges faster than the other counterparts. To simulate an alternative study with a modified guideline for X 1 , we sub-sampled ACTG 175, adjusting covariate distributions and assignments of tX 1 , X 2 u. Specifically, we evaluate a stochastic policy π 0 " tπ 0 px 1 | c 1 q, π 0 px 1 | c 2 qu for combining X 1 and X 2 based on C 1 , C 2 , with distribution P 0 representing a location with differing covariate distributions and treatment assignments. Further details are provided in Appendix D.2.We evaluated the AE est of all proposed estimators with and without noise (as described in the synthetic simulations). The AE plots are shown in Figs. (3c, 3d ). Results indicate that both the regression and DML estimators converge to the true policy effect faster under noisy conditions, whereas the PW estimator converges more slowly. However, DML does not consistently outperform at all sample sizes (see Fig. 3c ), as its error is influenced by the combined errors in the OM and PW estimators. Consequently, high error in the PW estimator may lead to increased error in the DML estimator. External validity: Project STAR We further examine policies on teacher-student ratios (i.e., class sizes) to improve academic achievement, using a semi-synthetic adaptation of the Project STAR dataset [40] . This longitudinal study evaluated the impact of teacher-student ratios on academic outcomes for students in kindergarten through third grade, with students randomized each year to one of three class size interventions. Here, we assess a 3-stage policy setting student-teacher ratios across Grades 0, 1, and 2, observing academic scores as intermediate outcomes, with baseline covariates (e.g., ethnicity, gender) and final academic scores at the end of Grade 3 as the primary outcome. To emulate data collected across different domains, we subsample using various probabilities to shift baseline covariate distributions, as done in ACTG 175 (see Appendix D.3 for details). We evaluated the PW, OM, and DML estimators across dataset sizes, plotting their absolute errors against the true effect of the candidate policy. Results, shown in Fig. 4 , mirror earlier experiments, with all estimators improving as sample size increases and DML showing faster convergence.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The data is publicly available and we have provided full details as to where to access the data and how to run the synthetic data generation pipeline. The code will not be open sourced at this moment but we believe to have provided sufficient details to reproduce our results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 This paper has considered the evaluation of the effectiveness of policies in settings where the available data is sampled from distributions that differ from the population in the target domain. We have illustrated this task with the problem of extrapolating the results of a clinical trial in both working examples and real-world scenarios to evaluate variations of the treatment in different populations.Our contributions are (1) introducing several identification criteria for the effectiveness of policies given experimental datasets from two or more domains and (2) developing doubly robust estimators for these settings that achieve fast convergence. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [TODO] 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We provided details where applicable. In our case, data splits, hyperparameters, optimizers, etc., are not significant for the implementation the method.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We report our results with error bars that represent 2 standard deviations from the mean.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [No] Justification: The paper does not involve crowd-sourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: Evaluating the impact of a policy using observational and experimental data under different conditions is a widespread challenge in various important decision-making fields. Formulations of this problem have appeared in the causality literature, but also in statistics, reinforcement learning, and epidemiology.Off-policy evaluation (OPE) aims to assess the performance of policies of interest using observational samples. In this line of research, [44] considers generalizing the effect of a policy under distribution shift. We build on this intuition, but instead seek to combine multiple (policy-)interventional data from source domains to learn the effect of policies of interest on the target domain. [1, 23] for instance used auxiliary datasets from multiple bandit instance, though they setting assume that the datasets are sampled from the same underlying populations and environments. Several authors have also considered transfer learning in off-policy learning in the context of bandits [46, 5] . Further, [22] addresses the problem of selection biases in observational data for off-policy learning.In the causal inference literature, combining multiple experimental studies to estimate a new causal effect is a task called generalized identification [30] . Recent progress has been made in developing corresponding estimators [26, 24] . However, these estimators are not applicable when our goal is to combine multiple policy interventional studies from source domains to estimate a causal effect in the target domain. Accordingly, [4, 29, 15] developed the notion of generalized transportability that aims to evaluate a causal effect on a target domain from multiple observational and / or interventional distributions from other source domains. In this line of research, our work relates closely to Correa and Bareinboim's identification algorithm for the effect of policies [15] . We similarly develop identification criteria that are conducive to efficient estimation from finite samples. In particular, our work focuses on the derivation of sample-efficient estimators for the policy effect of interest on the target domain.From this perspective, our work can be interpreted as a bridge between causal inference and off-policy evaluation [34] since we leverage formal theories in causal inference (e.g., generalized identification [29] , generalized transportability [15, 30] ) to solve off-policy evaluation problems efficiently from finite samples. There are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature use the backdoor adjustment to learn the Q value as a function of the state, to address confounding effects [41] . Meanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders [39] . Finally, some works have leveraged double negative controls for OPE [43] .Our work investigates the conditions under which policies may be estimated from multipe datasets collected under different conditions. In this work, we start from the assumption that causal and selection diagrams that are consistent with the underlying data generating systems of interest are available. In general, this requires domain knowledge and should be justified by prior knowledge or experiment. It is important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation, that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns mostly the second task. In higher-dimensional systems, the computational complexity of estimating the conditional expectations and density ratios that define our estimators could be a substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process.Since Theorem 1 is a special case for Theorem 4, we will only prove for Theorem 4.NoteEQUATIONThen,EQUATIONEQUATIONby leveraging the domain transfer for Y and adjustment for Y condition.For each P 0 π0 pw i | c, X pi´1q , w pi´1q q, P 0 π0 pw i | c, X pi´1q , w pi´1q q (4)EQUATIONagain, by leveraging the domain transfer condition for W i and adjustment condition for W i . This completes the proof.Since Theorem 2 is a special case for Theorem 5, we will only prove for Theorem 5. Throughout the proof, we will use C 1 :" C, X i :" tX i u for i " 1, ¨¨¨, m, and C i :" tW i´1 u for i " 2, ¨¨¨, m ´1. Also, we will sometimes use P 1 pC 1 q :" P 0 pCq, P i pC i | C pi´1q Y X pi´1q q for i ą 1 as P i´1 π i´1 pW i´1 | C, W pi´2q , X pi´1q q.Using the fact that ψ 0 " E P 0 rμ 1 0 s, we can write it asEQUATIONThen, we will claim and prove the following: Lemma 1 (Mixed Bias Property). Suppose µ i 0 , μi ă 8 and 0 ă π i 0 , πi ă 8 almost surely for i " 1, ¨¨¨, m. For i " 1, 2, ¨¨¨, m, define ϕ i ppC, W piq , X piq q; ω i , μi`1 , µ i q :" ω i tμ i`1 ´µi u,and μm`1 :" Y . Define ϕ 0 pC; μ1 q :" μ1 . For i " 0, ¨¨¨, m, define ϕ i 0 as ϕ i equipped with true nuisances, and φi as ϕ i equipped with estimated nuisances. Then, m ÿ i"0EQUATIONProof of Lemma 1. For i " m, ¨¨¨, 1 with μm`1 :" Y , defineµ i 0 rμ i`1 s :" E P i π i rμ i`1 | C, W piq , X piq s.Then,E P m πm rω m tμ m`1 ´μ m us `EP m πm rω m 0 μm s ´EP m πm rω m 0 µ m 0 rμ i`1 ss looooooooooomooooooooooon "ψ0" E P m πm rtω m ´ωm 0 utµ m 0 rμ i`1 s ´μ m us. For i " m ´1, ¨¨¨, 1,E P i π i rω i tμ i`1 ´μ i us `EP i π i rω i 0 μi s ´EP i π i rω i 0 µ i 0 rμ i`1 ss " E P i π i rtω i ´ωi 0 utµ i 0 rμ i`1 s ´μ i us.Also, the following holds:E P i`1 π i`1 rω i`1 0 μi`1 s " E P i π i rω i 0 µ i 0 rμ i`1 ss.Finally,E P 1 π 1 rω 1 0 μ1 s " E P 1 π 1 rμ 1 s.Therefore,m ÿ i"1 E P i π i rω i tμ i`1 ´μ i us `EP i π i rω i 0 μi s ´EP i π i rω i 0 µ i 0 rμ i`1 ss " m ÿ i"0 E P i π i r φi ´ϕi 0 s " m ÿ i"1 E P i π i rtω i ´ωi 0 utµ i 0 rμ i`1 s ´μ i us.RecallEQUATIONFrom Eq. ( 7),EQUATIONThen, the error ψ ´ψ0 can be decomposed intoEQUATION)DefineR i :" E D i ´P i π i rϕ i 0 s `1 L L ÿ ℓ"1 E D i ´P i π i r φi ℓ ´ϕi 0 s.Then, the error can be represented asψ ´ψ0 " m ÿ i"0 R i `1 L L ÿ ℓ"1 m ÿ i"0 E P i π i r φi ℓ ´ϕi 0 s. By Lemma 1, ψ ´ψ0 " m ÿ i"0 R i `1 L L ÿ ℓ"1 m ÿ i"0 E P i π i rtω i ℓ ´ωi 0 utµ i 0 ´μ i ℓ us.We will use the following results:Lemma 2 (Combining concentration inequalities). Suppose P pA k ą tq ď b k {t 2 for k " 1, ¨¨¨, K. Then,P ˜K ÿ k"1 A k ď tK ¸ě 1 ´1 t 2 K ÿ k"1 b k .Proof of Lemma 2. The event ř K k"1 A k ď tK includes the case where A k ă t for k " 1, ¨¨¨, K. Therefore,P ˜K ÿ k"1 A k ď tK ¸ě P pA 1 ď t and ¨¨¨and A K ď tq " 1 ´P pA 1 ą t or ¨¨¨or A K ą tq ě 1 ´K ÿ k"1 P pA k ą tq ě 1 ´K ÿ k"1 b k t 2 .Lemma 3 (Stochastic Equicontinuity). Let D iid " P . Let D " D 0 Ÿ D 1 , where n :" |D 0 |. Let f be a function estimated from D 1 . Then, in probability greater than 1 ´ϵ for any ϵ P p0, 1q,EQUATIONwhich implies thatE D0´P r| f ´f |s " O P ˜} f ´f } P ? n ¸.Proof of Lemma 3. This proof is from [28, Lemma 2] . Since f is a function of D 1 , we will denote fD1 . Define a following random variable of interest:X :" E D0´P r fD1 ´f s.Then, the conditional expectation of X given D 1 is zero, sinceE P « 1 n n ÿ i"1 fD1 pV i q ˇˇˇD 1 ff " 1 n n ÿ i"1 E P r fD1 pV i q | D 1 s " 1 n n ÿ i"1 E P r fD1 pVq | D 1 s " E P r fD1 pVq | D 1 s,where the third equality holds by the independence of D 0 and D 1 . Therefore,E P rX | D 1 s " E P rE D0´P r fD1 ´f s | D 1 s " E P rE D0 r fD1 ´f s | D 1 s ´EP rE P r fD1 ´f s | D 1 s " E P rE P r fD1 ´f s | D 1 s ´EP rE P r fD1 ´f s | D 1 s " 0.Also,V P rX | D 1 s " V P rE D0´P r fD1 ´f s | D 1 s " V P rE D0 r fD1 ´f s | D 1 s " 1 n V P r fD1 ´f | D 1 s ď 1 n } fD1 ´f } 2 P .By applying the (conditional-) Chevyshev's inequality,P p|X ´EP rX | D 1 s| ě t | D 1 q ď 1 t 2 V P rX | D 1 s ď 1 nt 2 } fD1 ´f } 2 P .Then,P p|X| ě tq " P p|X ´EP rX | D 1 s| ě tq " E P pD1q rP p|X ´EP rX | D 1 s| ě t | D 1 qs ď 1 nt 2 } fD1 ´f } 2 P .In other words, X ă t in probability greater than 1´1 nt 2 } fD1 ´f } 2 P . If t " } fD 1 ´f } P ? nϵ , then X ă } fD 1 ´f } P ? nϵin the probability greater than 1 ´ϵ for any ϵ P p0, 1q.Here, we will study the finite sample behavior ofm ÿ i"0 R i :" m ÿ i"0 E D i ´P i π i rϕ i 0 s `1 L L ÿ ℓ"1 m ÿ i"0 E D i ℓ ´P i π i r φi ℓ ´ϕi 0 s.By Chevyshev's inequality, P r˜ˇˇE D i ´P i π i rϕ i 0 s ˇˇą t ρ i,0 a |D i | ¸ă 1 t 2 ,or equivalently, P r´ˇˇE D i ´P i π i rϕ i 0 s ˇˇą t ¯ă 1 t 2 ρ 2 i,0 |D i | .By Lemma 2, P r˜m ÿ i"0 ˇˇE D i ´P i π i rϕ i 0 s ˇˇď pm `1qt 1 ¸ě 1 ´1 t 2 1 m ÿ i"0 ρ 2 i,0 |D i | .By Lemma 3, P r´ˇˇE D i ℓ ´P i π i r φi ℓ ´ϕi 0 s ˇˇą t 2 ¯ď 1 t 2 2 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ |.By Lemma 2,P ˜1 L L ÿ ℓ"1 m ÿ i"0 ˇˇE D i ℓ ´P i π i r φi ℓ ´ϕi 0 s ˇˇď pm `1qt 2 ¸ě 1 ´1 t 2 2 L ÿ ℓ"1 m ÿ i"0 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ | . Choose t 1 :" c 2 ϵ ř m i"0 ρ 2 i,0 |D i | and t 2 :" d 2 ϵ ř L ℓ"1 ř m i"0 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ |. Then, with a probability greater than 1 ´ϵ,m ÿ i"0 R i ď pm `1q ¨g f f e 2 ϵ m ÿ i"0 ρ 2 i,0 |D i | `g f f e 2 ϵ L ÿ ℓ"1 K ÿ i"1 } φk ℓ ´ϕk 0 } 2 P k |D k ℓ | ‚ " pm `1q c 2 ϵ ¨g f f e m ÿ i"0 ρ 2 i,0 |D i | `g f f e L ÿ ℓ"1 m ÿ i"0 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ | ‹ ‚.We will use the following result:Proposition 1 (Berry-Esseen's inequality [7, 19, 38] ). Suppose D " tX 1 , ¨¨¨, X n u are independent and identically distributed random variables with E P rX i s " 0, E P rX 2 i s " σ 2 and E P r|X i | 3 s " κ 3 . Then, for all x and n, ˇˇˇP ˆ?n σ 0 E D rXs ă x ˙´Φpxq ˇˇˇď 0.4748κ 3 σ 3 ? n .By Lemma 3, P rEQUATION)By Lemma 2,EQUATION)Define∆ i :" g f f e 1 ϵ L ÿ ℓ"1 } φi ℓ ´ϕi 0 } 2 P i π i |D i ℓ |.With a probability greater than 1 ´ϵ,1 L L ÿ ℓ"1 ˇˇE D i ℓ ´P i π i r φi ℓ ´ϕi 0 s ˇˇw .p 1´ϵ ď ∆ i .DefineA i :" E D i ´P i π i rϕ i 0 s B i :" 1 L L ÿ ℓ"1 E D i ℓ ´P i π i r φi ℓ ´ϕi 0 s C i :" 1 L L ÿ ℓ"1 ˇˇE D i ℓ ´P i π i r φi ℓ ´ϕi 0 s ˇˇ.Then, R i " A i `Bi . Then, P r pR i ă xq (16)EQUATION" P r pA i ă x ´Bi q (18)EQUATION)Then,|P r pA i ă x `∆i q ´Φpxq| (21)" |P r pA i ă x `∆i q ´Φpx `∆i q `Φpx `∆i q ´Φpxq| (22) ď |P r pA i ă x `∆i q ´Φpx `∆i q| `|Φpx `∆i q ´Φpxq| (23)EQUATION)This completes the proof. ■By Cauchy-Schwartz' inequality,EQUATION)Given assumption, the upper bound in Eq. ( 18) converges at 1{ a |D i ℓ | rate. Therefore, R i converges in distribution to normalp0, ρ 2 i,0 q.Codes corresponding to simulations are submitted as supplementary materials.We define the following SCM. First,U XW , U X1,X2 , U X2,W , U X2,Y , U C1,1 , U C1,2 , U C2,1 , U C2,2 , U W , U Y " normalp0, 1q. Then, C 1 :" f C1 pSq " 0.25SU C1,1 `0.1S `UC1,1 C 2 :" f C2 pSq " 0.25SU C2,1 `0.1S `UC2,2 X 1 :" f X1 pC 1 , C 2 , Sq " Bernoulipπ 1,S pC 1 , C 2 qq W :" f W pC 1 , C 2 , X 1 , U X1,W , Sq " sigmoidp0.25SU W `0.5U X1,W `3X 1 `0.5pC 1 `C2 qq X 2 :" f X2 pX 1 , W, C 1 , C 2 , Sq " Bernoulipπ 2,S pC 1 , C 2 qq Y :" f Y pC 1 , C 2 , X 1 , X 2 , W, U X2,Y , Sq " sigmoidp0.5pC 1 `C2 q `2pX 1 `X2 q ´2 ´0.5W `0.1U X2,Y `0.25SU W q.Also, for S ‰ 0, π 1,0 " sigmoidpC 1 `C2 ´2q π 1,S " sigmoidp0.5pC 1 `C2 q ´1q π 2,0 " sigmoidp0.5pC 1 `C2 q `2p2X 1 ´1q ´0.5W `1q π 2,S " sigmoidpC 1 `C2 `2X 1 ´1 `0.5W ´1q. D.1.2 Synthetic Simulations for Fig. 3b We define the following SCM. First,U X1,X2 , U X2,X3 , U W1,X1 , U W1,X2 , U W2,X2 , U W2,X3 , U C1,C2 , U C2,C3 , U X3,Y , U C1,1 , U C1,2 , U C2,1 , U C2,2 , U C3,1 , U C3,2 , U W1 , U W2 , U Y " normalp0, 1q. Then, C1 :" fC 1 pSq " 0.25SUC 1,1 `0.1S `UC 1,2 `UC 1 ,C 2 C2 :" fC 2 pSq " 0.25SUC 2,1 `0.1S `UC 2,2 `UC 1 ,C 2 `UC 2 ,C 3 C3 :" fC 3 pSq " 0.25SUC 3,1 `0.1S `UC 3,2 `UC 2 ,C 3 X1 :" fX 1 pC1, C2, Sq " Bernoulipπ1,SpC1, C2qq W1 :" fW 1 pC, X1, UW 1 ,X 1 , UW 1 ,X 2 , Sq " sigmoidp0.25SUW 1 `0.5pC1 `C2 `C3q ´1 `3X1 `0.5pUW 1 ,X 1 `UW 1 ,X 2 q `Sq X2 :" fX 2 pX1, W1, C1, C2, C3, Sq " Bernoulipπ2,SpX1, W1, C1, C2, C3, Sqq W2 :" fW 2 pC, X1, X2, W1, UW 2 ,X 2 , UW 2 ,X 3 , Sq " sigmoidp0.25SUW 2 `0.5pC1 `C2 `C3q ´1 `3pX1 `X2q `0.5pUW 2 ,X 2 `UW 2 ,X 3 q `Sq X3 :" fX 3 pX1, X2, W1, W2, C1, C2, C3, Sq " Bernoulipπ3,SpX1, X2, W1, W2, C1, C2, C3, Sqq Y :" fY pC, X, W, UX 3 ,Y , Sq " sigmoidp0.5pC1 `C2 `C3q `2pX1 `X2 `X3q ´3 ´0.5pW1 `W2q `0.1UX 3 ,Y `0.25SUW `Sq.Also, for S ‰ 0, π 1,0 " sigmoidpC 1 `C2 `C3 ´2q π 1,S " sigmoidp0.5pC 1 `C2 `C3 q ´1q π 2,0 " sigmoidp0.5pC 1 `C2 `C3 q `2p2X 1 ´1q ´0.5W `1q π 2,S " sigmoidpC 1 `C2 `C3 `2X 1 ´1 `0.5W ´1q π 3,0 " sigmoidp0.25pC 1 `C2 `C3 q `p2X 1 ´1q ´0.25W 1 `1 `p2X 2 ´1q ´0.25W 2 `1q π 3,S " sigmoidp0.5pC 1 `C2 `C3 q `2p2X 1 ´1q `0.25W 1 ´1 `2p2X 2 ´1q `0.25W 2 ´1q.To provide empirical evidence of policy estimation in a real-world setting, we revisit the ACTG 175 randomized clinical trial from 1994 conducted on patients from the United States and Puerto Rico [21] . It investigated the effectiveness of different therapies for reducing CD4 T cell counts in individuals with HIV (selected subject to various inclusion criteria). In the study, individuals were randomly assigned to two different treatments X 2 P t0, 1u, and a record was made on whether a previous anti-retroviral drug had been administered X 1 P t0, 1u prior to the start of the trial. Patient demographics C 1 , C 2 including gender, age, weight, among others, were collected, and CD4 T cell count were measured at treatment time W , and again 20 weeks after treatment initialization Y , the outcome of the analysis. To simulate a second study with a different guideline for anti-retroviral drug administration, we considered a sub-sampled version of ACTG 175 in which covariate distributions as well as the assignment of X 1 , X 2 were modified.ACTG 175 is an experimental study in which X 2 has been randomized and X 1 follows a baseline, unknown, stochastic policy π 2 : Ω C2 ˆΩX1 Ñ r0, 1s assumed to depend on study-specific features C 2 such a patient's Karnofsky score and symptomatic indicators (both normalized to lie in the r0, 1s interval). Samples of variables C 1 , C 2 , W, X 1 , X 2 , Y therefore follow a distribution P 2 randpX2q,π2 pC 1 , C 2 , W, X 1 , X 2 , Y q. The suffix "randpX 2 q" denotes a policy that randomizes X 2 , i.e. X 2 " Bernp0.5q. We generate a data sample from a second domain (S " 1) following the marginalized distribution P 1 π1 pC 1 , W, X 1 q, mimicking a simple stochastic guideline on X 1 in which π 1 :" π 1 px 1 " 1 | c 1 q " 1{p1 `expt´c 11 ´c12 ´c13 uq. Higher values of C 1 (taken to be normalized measurements of weight, height, and age) lead to higher likelihood of treatment. We achieve this dataset by sampling according to a re-weighted version of the ACTG 175 trial. In particular, we collect data from P 1 according to,X 1 W X 2 C 1 C 2 Y π 2 (a) G ACTG " G 2 π 2 X 1 W X 2 C 1 C 2 Y π 1 (b) G 1 π 1 X 1 W X 2 C 1 C 2 Y (c) G 0 X 1 W X 2 C 1 C 2 Y S C1 S C2 S Y π 0 π 0 (d) G ∆ 0,1 π 0 X 1 W X 2 C 1 C 2 Y S C1 S C2 S Y π 0 π 0 (e) G ∆ 0,2 π 0P 1 π1 pC 1 , W, X 1 q :" P 2 π2 pC 1 , W, X 1 q π 1 pX 1 | C 1 qP 1 pC 1 q P 2 π2 pX 1 | C 1 qFor this example, we consider evaluating a stochastic policy π 0 " tπ 0 px 1 | c 1 q, π 0 px 1 | c 2 qu that combines the drugs X 1 , X 2 according to a stochastic policy for X 1 based on weight, height, and age, pC 1 q and for X 2 based on a patient's Karnofsky score and symptomatic indicators pC 2 q. In particular,EQUATIONThe policy π 0 is considered to be implemented on a patient population located in a different location that are know to have a differing covariate distribution P 0 pC 1 , C 2 q to that observed in ACTG 175 and the second study, among other discrepancies. We assume that the SCM generating this experimental study follows the causal graphs in Fig. 5 .The target population under π 0 is then given byP 0 π0 pC 1 , C 2 , W, X 1 , X 2 , Y q :" P 2 randpX2q,π2 pC 1 , C 2 , W, X 1 , X 2 , Y q P 0 pC 1 , C 2 qπ 0 pX 1 | C 1 qπ 0 pX 2 | C 2 q P 2 pC 1 , C 2 qP 2 π2 pX 1 | C 1 qP 2 randpX2q,π2 pX 2 qWe limit all datasets to approximately 2000 samples as this is the size of the ACTG 175 trial. The ground truth target effect E P 0 π 0 rY s is evaluated by taking the empirical mean of Y in the sample of data collected from P 0 with the procedure above.We describe in this section additional experimental details on the Project STAR study 4 . This study investigated the impact of teacher/student ratios on academic achievement for kindergarten through third-grade students. Project STAR was a four-year longitudinal study where students were randomly assigned to one of three interventions with different class sizes each year, following different randomization procedures. The causal diagram we assume for this setting is provided in 4 The dataset is publicly accessible from the R data repository: https://search.r-project.org/CRAN/refmans/AER/html/STAR.html. Fig. 6 . Bi-directed arcs denote unobserved confounding in the observational regime (when student are observed in a particular class size rather than forced to join a particular class size). Specifically, we consider the evaluation of a 3-stage stochastic policy,π 0 " tπ 0 px 1 | c 1 q, π 0 px 2 | c 2 , x 1 , w 1 q, π 0 px 3 | c 3 , x 2 , w 2 qu,where, π 0 px 1 | c 1 q :" 1{p1 `exptc 11 `c12 ´2uq π 0 px 2 | c 2 , x 1 , w 1 q :" 1{p1 `expt0.5pc 21 `c22 q `2p2x 1 ´1q ´0.5w 1 `1uq π 0 px 3 | c 3 , x 2 , w 2 q :" 1{p1 `expt0.5pc 31 `c32 q `p2x 1 ´1q ´0.25w 1 `p2x 2 ´1q ´0.25w 2 `1uqThese policies determine the student-teacher ratio X 0 , X 1 , X 2 , taking values "regular" or "small", across three different grades, namely Grade 0 (Kindergarten), Grade 1 and Grade 2. C refers to a two-dimensional demographic variable encoding gender and ethnicity, converted to binary and categorical variables respectively. (To avoid clutter, in Fig. 6 we use C " C 1 " C 2 " C 3 .) W 1 , W 2 are intermediate school outcomes that include the sum total of an individual's reading score and math score in grades 0 (Kindergarten) and 1 respectively. Y is the outcome of interest and represents total reading score and math score in grade 2.To mimic the setting where data at different stages was collected from different domains, we subsample the dataset using different sets of probabilities to induce differences in the distributions of baseline covariates. In particular, we fix the dataset in the target domain (S " 0) to the distribution observed in the study and sub-sample according to different probabilities to create datasets for domains S " 1, S " 2, and S " 3, as follows.We generate a sample of data from a first source domain (S " 1) following the marginalized distribution P 1 π1 pC 1 , W 1 , X 1 q, where π 1 :" π 1 px 1 " 1 | c 1 q " 1{p1 `expt0.5pc 11 `c12 q ´1uq defines the probability for the student-teacher ratio variables in Kindergarten in domain S " 1. We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 1 according to, P 1 π1 pC 1 , W 1 , X 1 q :" P 0 pC 1 , W, X 1 qπ 1 pX 1 | C 1 qP 1 pC 1 q P 0 pX 1 | C 1 qwhere P 1 pc 1 q " 0.3 if c 11 " 1, c 12 " 1 and P 1 pc 1 q " 0.7 otherwise.We generate a sample of data from a second source domain (S " 2) following the marginalized distribution P 2 π2 pC 2 , W 1 , W 2 , X 1 , X 2 q, where π 2 :" π 2 px 2 " 1 | c 2 , x 1 , w 1 q " 1{p1 `expt0.5pc 21 c22 q `2x 1 ´1 ´0.5w 1 ´1uq defines the probability for the student-teacher ratio variables in grade 1 in domain S " 2. We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 2 according to,P 2 π2 pC 2 , W 1 , W 2 , X 1 , X 2 q :" P 0 pC 2 , W 1 , W 2 , X 1 , X 2 q π 2 pX 2 | C 2 , X 1 , W 1 qP 2 pC 2 q P 0 pX 2 | C 2 , X 1 , W 1 qwhere P 2 pc 2 q " 0.7 if c 21 " 1, c 22 " 1 and P 2 pc 2 q " 0.3 otherwise.We generate a sample of data from a third source domain (S " 3) following the marginalized distribution P 3 π3 pC 3 , W 1 , W 2 , X 1 , X 2 , X 3 q, where π 3 :" π 3 px 3 " 1 | c 3 , x 1 , w 1 , x 2 , w 2 q " 1{p1 èxpt0.5pc 31 `c32 q `2p2x 1 ´1q ´0.25w 1 `p2x 2 ´1q `0.25w 2 ´1uq defines the probability for the student-teacher ratio variables in grade 3 in domain S " 3. We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 3 according to,P 3 π3 pC 3 , W 1 , W 2 , X 1 , X 2 , X 3 , Y q :" P 0 pC 3 , W 1 , W 2 , X 1 , X 2 , X 3 , Y q π 3 pX 3 | C 3 , W 1 , W 2 , X 1 , X 2 , X 3 qP 3 pC 3 q P 0 pX 3 | C 3 , X 1 , W 1 , X 2 , W 2 qwhere P 3 pc 3 q " 0.5 if c 31 " 1, c 32 " 1 and P 3 pc 3 q " 0.5 otherwise. • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.X 1 W 1 X 2 C 1 C 2 C 3 W 2 X 3 Y• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: We have read the guidelines and we do not think that our work presents any notable concerns.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Thm. 1 remains valid for a mixture of discrete and continuous W , C, and X. For these cases, sums can be appropriately replaced by Lebesgue integrals. However, we continue to use summation notation in our explanation to keep the presentation of the identification result straightforward. 


Decision: All the reviewers agree the problem tackled in the paper, offline policy evaluation in transfer learning setup, is important. There were concerns about the lack of real data evaluation, the assumptions / settings of the theory and missed related works. In addition the paper seems to suffer from presentation.

Most of the concerns have been addressed by the authors’ rebuttal and the reviewers are leaning towards accepting this paper.

I would still urge the authors to improve the presentation and make the paper widely accessible to neurips community.

Reviews: review: paper_summary: paper_summary: This work presents a method for evaluating effectiveness of policies across multiple domains using a new graphical criteria and estimators  by combining data from multiple experimental studies. The authors report error analysis of the proposed estimators that gives provides fast convergence guarantee, and additionally share simulation results as well as empirical verification on real data.
strength_weakness: strength_weakness: The the proposed method seems to have important potentials in real world applications, though the majority of the paper has been dedicated to theoretical analysis and the empirical evaluations are very limited.
There is little discussion on how this could impact realworld problems, and there is not a detailed analysis on the empirical results of the results on ACTG175 dataset.

From figure 3.c, it can be seen a contradictory message that the proposed DML method does not perform better than OM on all datasets, though there is no discussion on that. 

also authors state that "simulations on real and synthetic data are provided for illustration purposes only." which brings the question to what extend this approach is useful in real world scenarios.
questions: questions: - Please provide more empirical analysis on the proposed method ideally on more real-world datasets if possible.

- Please discuss the results on ACTG175 datasets and the different pattern observed in Fig3.c.

- Provide more empirical evidence on real-world data.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper studies off-policy evaluation in a transfer learning setting with multiple source datasets collected from observational and/or randomized studies. The objective is to evaluate the effect of a target policy on a possibly different target population. To achieve this, the author(s) assume at each time point, at least one source population shares the same distribution of the dynamic variable given the historical covariate-treatment pair as the target population. This ensures the identifiability of the target population's policy value, even no source population perfectly matches its entire distribution over time. The author(s) further propose doubly robust estimators, investigate their theoretical properties and finite sample performance.
strength_weakness: strength_weakness: I have two major concerns, regarding the technical assumptions and numerical experiments, along with several moderate comments. I detail them one by one below:

**Assumptions**. The assumptions might appear overly realistic:

* **Same number of studies to horizon**. In addressing a $T$-horizon off-policy evaluation (OPE) problem, the author(s) assume that there are exactly $T$ studies, each corresponding to a time point $t$, where exactly one study per time point shares the same outcome distribution (the distribution of $Y_T$ if $t=T$ and that of $W_{t+1}$ otherwise) with the target population at that time. These settings seem unrealistic in practical scenarios. It would be beneficial for the author(s) to provide examples of real applications that validate this assumption. The current analysis using the ACTG study seems artificial; I will elaborate on this concern in more detail later.
* **Knowledge of matching source population**. Additionally, the author(s) seem to require prior knowledge of which source population matches the target population’s outcome distribution at each time point. In practical scenarios, while we may have access to multiple studies, it is typically unknown which one aligns with the target's outcome distribution at each time point. A more realistic approach would adaptively learn which studies are similar to the target distribution at each time based on the data. This adaptive learning scenario, in my opinion, would better fit real-world applications.
* **Identical distributions**. Although the outcome distributions between the source and target populations may be similar at each time, they are not necessarily identical. Even if the distributions are not exactly the same, as long as the differences are minimal, it is reasonable to use source data for transfer learning remains. Recent studies have addressed such distributional shifts using regularization or adaptive weighting: 
    - https://arxiv.org/pdf/2112.09313
    - https://arxiv.org/pdf/2111.15012
    - https://arxiv.org/pdf/2406.00317

**Numerical experiments**. The experiments are overly simplified:

* **Single-horizon settings.** While the paper studies multi-horizon dynamic treatment regimes, the simulations are conducted in a single stage setting. I would suggest to use D4RL benchmark datasets or OpenAI Gym environments to evaluate the proposed methodologies in multi-stage studies.
* **Lack of competing methods.** There are some naive estimator to consider in these transfer learning settings. However, the author(s) did not include them in the experiments. For instance, one can ignore the differences in the outcome distributions and assume the multiple datasets come from a single population, with a mixture of multiple behavior policies that generates the action. 
* **Real data**. The ACTG dataset is generated under a single population. To evaluate the proposed methodology, the author(s) manually created a second population and a target population. It would be better if a real "multi-source" dataset could be used.

**Related works**. 
* Under the identical distribution assumption, this paper mainly considers settings with covariate shift. There are some recent works that studied similar problems, although in single-stage settings, e.g.,  https://onlinelibrary.wiley.com/doi/epdf/10.1111/biom.13583. 
* In the related work section, the author(s) argued that their work can be viewed as a bridge between causal inference and OPE by  leveraging formal theories in causal inference to solve OPE problems. In that sense, there are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature uses the backdoor adjustment to learn the Q- or value as a function of the state, to address the confounding effects (see e.g., https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf). Meanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders (https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2110878). Finally, some works have leveraged double negative controls for OPE (https://cdn.aaai.org/ojs/6590/6590-13-9815-1-10-20200520.pdf). 

**Typo**: $Z$ is used to denote the dynamic covariate in Section 2. However, this notation has been replaced with $W$ starting from Section 3.
questions: questions: Would it be possible to establish the semiparametric efficiency of the proposed estimators?
limitations: limitations: The limitations have been discussed in the appendix.
paper_summary: paper_summary: The paper develops novel graphical criteria and estimators for evaluating the effectiveness of various policies by integrating data from multiple experimental studies.  Through theoretical analysis and empirical validation via simulations, the paper demonstrates the robustness and fast convergence of the proposed estimators.
strength_weakness: strength_weakness: The paper is difficult to follow. It is very hard to understand. Some grammatical mistakes and confused notations, e.g.,
1. line 71, ‘variable $\mathbf{Z}$ For example’ -> ‘variable $\mathbf{Z}$. For example’
2. unify the notations if necessary, try to avoid writing $\bigtriangleup_{ij}$ and $\bigtriangleup_{i, j}$ simultaneously.
For the experimental section, please give detailed descriptions on the simulated experiment and the empirical experiment (if the descriptions are too long, please move to the appendix), which allows reader to understand the advantages of the proposed framework.
For the theoretical section, the authors first give study when combining two experiments, followed by the study when combining multiple experiments. Honestly, the materials in the two studies are similar. Authors can unify the two studies: presenting the study when combining multiple experiments followed by the two examples (example 2 and example 3). This can release more space to discuss the core components.
questions: questions: Consider example 3. The proposed framework allows researchers to evaluate a policy from the three experiments. Suppose that all the datasets are synthetic together such that the resulting data is treated as a single dataset obtained from one experiment; probably, we can evaluate a policy based on the synthetic dataset as well. Could you conduct a study that compares the two policy evaluation methods?

In some situations, it is necessary to evaluate a policy when the size of the multiple treatments is large. Consider example 3 again, but this time, we consider all the big cities in the US instead of three cities in the US. The computation complexity increases, obviously. Are there any efficient methods to handle the relevant problem?

The adjustment criterion assumes that the distribution is invariant across various experiments. To my realization, the distributions vary even in the examples presented in the paper. How does the proposed methodology handle situations where the underlying assumptions are violated?
limitations: limitations: The paper focuses on combining two experiments for policy evaluation only. It would be better if the authors can discuss the optimal policy when combining two experiments.
paper_summary: paper_summary: This paper studies how to evaluate policies where source and target sites have distribution shifts. The authors introduce identification criteria for the effectiveness of policies, and develop doubly robust estimators that achieves fast convergence. The results are also generalized to multiple source datasets. Simulation results are provided to show the effectiveness of the method.
strength_weakness: strength_weakness: I am not an expert in this field and not familiar with causal inference. I find no major weaknesses. Some minor weaknesses are on the empirical evaluations. For example, the non-synthetic experiments are only conducted on ACTG 175 clinical trial dataset. Experimenting on other different datasets will enhance the empirical results.
questions: questions: See weaknesses part.
limitations: limitations: N/A

Comments: title: Post rebuttal comment
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
comment: I would like to thank the author(s) for their responses. I would keep my score now and vote for acceptance.
I thank the authors for the provided response. In light of the additional experiments, I have updated my scores accordingly.
The discussion period is nearing its end, with just about a day remaining. Could you please check and confirm if our rebuttal has addressed your concerns and comments?
The discussion period is nearing its end, with just about a day remaining. Could you please check and confirm if our rebuttal has addressed your concerns and comments?
The discussion period is nearing its end, with just about a day remaining. Could you please check and confirm if our rebuttal has addressed your concerns and comments?
The discussion period is nearing its end, with just about a day remaining. Could you please check and confirm if our rebuttal has addressed your concerns and comments?