Title: Efficient Policy Evaluation Across Multiple Different Experimental Datasets
Venue: NeurIPS 2024
Accepted: yes
Keywords: dynamic treatment regime

data fusion

off-policy evaluation

transportability
Abstract: Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from the same distribution. This assumption is often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability of the effectiveness of policies in a target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies (e.g., conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.

Efficient Policy Evaluation Across Multiple Different Experimental Datasets

Abstract

Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from the same distribution. This assumption is often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability of the effectiveness of policies in a target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies ( e.g. , conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.

1 Introduction

In the empirical sciences, conclusions on the effect of actions or policies is often supported by evidence drawn from prior observations and experiments. The conditions under which such inferences can be formally justified can be traced back (in part) to Campbell, Stanley and Cook [10, 11, 14]. They argued for a basic dichotomy in the kinds of questions that scientists seek to answer from experimental data. On the one hand asking whether ' in fact, the experimental stimulus made some significant difference in this specific instance? ', and on the other hand asking ' to what populations, settings, and treatments can this effect be generalized? ' [10, p. 297]. These inferences have since been labelled as internal validity and external validity , respectively.

External validity is concerned with the extent to which findings from one population can be 'reprocessed', or 're-calibrated' so as to circumvent population differences and produce valid generalizations in a target population where experiments cannot be performed (e.g., outside the laboratory, different domains, etc.). The validity of these inferences will necessarily be contingent on a careful analysis to ascertain the commonalities and differences between domains as, for example, if the target domain is completely arbitrary generalization is impossible. In the causal transportability literature, the basis for generalization (also called transportability) is justified by the stability and invariance of the causal mechanisms shared across populations and domains [20, 32]. Several graphical characterizations exist to delineate the conditions under which transportability is possible, with recent algorithms proposing solutions for general instances of the external validity task combining observational and experimental distributions under partial observability [36, 2, 3, 16, 30].

These algorithmic solutions express a target policy effect in terms of the observational and experimental source distributions. Still, then one needs to go further and estimate the resulting expression from finite samples. In practice, with a finite number of samples and potentially high-dimensional covariates, estimating causal expressions is quite challenging. Effective estimators have been developed for specific settings, starting with doubly-robust estimators for functionals given by the backdoor criterion [13, 37, 9, 45], and recently extended to cover general identification scenarios with observational and experimental samples [25, 26, 8]. These techniques also find parallels across other related disciplines, such as reinforcement learning where re-weighting [42, 31], outcome modelling [6], and doubly-robust estimation [18], are common for evaluating the effect of policies to overcome shifts in the behaviour policy. Recently, [44] and [22] have considered policy evaluation under covariate shift and selection bias, a special case of the external validity problem with a given graph. Despite their generality, existing estimators still only cover a limited portion of realistic scientific inferences. In particular, existing methods are not applicable in settings where datasets are collected in different domains.

We consider the generalization of causal claims from observational and experimental data through the task of policy evaluation . The target for inference is E P 0 π r Y s where P 0 π symbolizes the distribution of data in a target domain (indexed as 0 ) in which a hypothetical policy of interest π (also known as dynamic treatment regimes [33] or soft interventions [16]) has been implemented. The question then becomes how to identify and estimate E P 0 π r Y s , given finite samples from multiple observational and experimental data (e.g., P i π i , a source domain indexed by i in which experimental policy is π i that may differ with π 0 ) collected under different settings and structural assumptions, encoded in causal diagrams. We aim to bridge the gap between identification and estimation to solve general instances of external validity. Our contributions are twofold:

Sec. 3: We develop nonparametric identification criteria (Thm. 1) to determine whether the effect of a policy may be expressed through an adjustment formula from two separate distributions induced by policy interventions, collected from different populations. Based on this formulation, we develop a multiply robust estimator (Thm. 3) that enjoys multiply robustness against model misspecification and bias.

Sec. 4: We generalize these identification criteria (Thm. 4) and propose a general multiply-robust estimator (Thm. 6) applicable for the evaluation of policies from multiple source datasets.

1.1 Preliminaries

We use bold letters ( X ) to denote a random vector and X a random value. Each random vector is represented with a capital letter ( X ) and its realized value with a small letter ( x ). Given a set X ' t X 1 , ¨ ¨ ¨ , X n u , we denote X p i q : ' t X 1 , ¨ ¨ ¨ , X i u . For a discrete vector X , we use ✶ x p X q to represent the indicator function such that ✶ x p X q ' 1 if X ' x ; ✶ x p X q ' 0 otherwise. For comprehensibility, we use P p v q to denote a probability at V at v for discrete/continuous random variables V . In similar, we use ř z for Z Ď V for the summation/integration over a mixture of discrete/continuous random variables Z For example, we write the back-door adjustment as ř z E P r Y | x, z s P p z q even when Z is a mixture of discrete/continuous variables. We use E P r f p V qs : ' ř v f p v q P p v q for a function f . For a sample set D : ' t V p i q : i ' 1 , ¨ ¨ ¨ , n u where V p i q denotes the i th samples, we use E D r f p V qs : ' p 1 { n q ř n i ' 1 f p V p i q q . We use } f } P : ' a E P rt f p V qu 2 s . If a function p f is a consistent estimator of f having a rate r n , we use p f ´ f ' o P p r n q . We say ˆ f is L 2 -consistent if } ˆ f ´ f } P ' o P p 1 q . We use p f ´ f ' O P p 1 q if p f ´ f is bounded in probability, and p f ´ f ' O P p r n q when p f ´ f is bounded in probability at rate r n .

We use Structural Causal Models (SCMs) as our framework [35]. An SCM M is a quadruple M ' x U , V , P p U q , F y . U is a set of latent variables following a joint distribution P p U q . V is a set of observable variables whose values are determined by functions F ' t f V i : V i P V u such that V i Ð f V i p pa V i , u V i q where PA i Ď V and U V i Ď U . Each SCM M induces a distribution P p V q and a causal graph G in which directed edges from every variable in PA i to V i exist. Dashedbidirected arrows encode correlated latent variables.

2 Policy Evaluation Integrating Multiple Experimental Datasets

We investigate the sequential decision-making setting concerning a set of actions X , a series of dynamic covariates Z , a series of static covariates C and an outcome variable of interest Y in an SCM M . A policy vector π : ' t π i u over actions X ' t X 1 , ¨ ¨ ¨ , X m u is an ordered set of decision rules for each X i P X . Actions are selected according to a topological ordering X 1 ă ¨ ¨ ¨ ă X K over time. Each action X i is potentially associated with a set of prior static and dynamic covariates, for example, the decision rule for X k could be defined as x k ' π p¨ | z p k q , x p k ´ 1 q , c p k q q . Every π p X k | Z p k q , X p k ´ 1 q , C p k q q is a probability distribution mapping from domains of the set of inputs t Z p k q , X p k ´ 1 q , C p k q u to the domain of actions X k . The implementation of a policy π in M induces an intervened model M π , that sets values of every X P X to be decided by the policy π , replacing the functions t f X , X P X u that would normally set its value. We denote a distribution induced by M π as P π . Now, we fix the notion of the policy evaluation as follows:

Definition 1 ( Policy evaluation [41]) . The policy evaluation is to predict the effectiveness of a policy vector π on an outcome Y in an target SCM M 0 ; i.e., ψ 0 : ' E P 0 π r Y s .

Difficulties in estimating E P 0 π r Y s comes from that the distribution or samples from P 0 π are generally not available. These discrepancies can be formalized under the rubric of SCMs as follows. In the most general setting, an investigator might leverage multiple source domains t M 1 , M 2 , . . . , M K u over V that entail distributions P : t P 1 , P 2 , . . . , P K u . Data or samples from these distributions may be available under different behaviour policies, e.g. , π 1 , π 2 , . . . , π K , depending on the study or data collection protocol implemented in each domain (that might include an observational regime, i.e. no policy implemented). To ground the policy evaluation problem, we define graphical tools to capture commonalities and discrepancies across domains.

Definition 2 ( Domain discrepancy [29]) . For every pair of SCMs M i , M j ( i, j P t 0 , 1 , 2 , . . . , K u ) defined over V , the domain discrepancy set ∆ ij Ď V is defined such that for every V P ∆ ij there might exist a discrepancy between f M i V ‰ f M j V , or P M i p u V q ‰ P M j p u V q .

Definition 3 ( Selection diagram [29]) . The selection diagram G ∆ ' t G j u j Pt 0 , 1 , 2 ,...,T u Ť t G ∆ 0 j u j Pt 1 , 2 ,...,T u is a graph constructed from G i ( i P t 0 , 1 , 2 , . . . , T u ) by adding the selection node S ij to the vertex set, and adding the edge S ij Ñ V for every V P ∆ ij .

∆ i,j locates the mechanisms where structural discrepancies between two domains are suspected to take place. V R ∆ i,j represents the assumption that the mechanisms for V are invariant across the two domains. The induced selection diagram is a parsimonious representation of these constraints. The following example illustrates these notions.

Example 1 ( External validity under covariate shift ) . A common instance of the external validity problem in the literature considers the evaluation the effect of a policy π : Ω C ˆ Ω X Ñr 0 , 1 s for assigning a treatment X P t 0 , 1 u , subject to shift in the distribution of covariates C . For this example, let source and target domains M : t M 1 , M 0 u over V ' t C, X, Y u , U ' t U XY , U C u be defined as follows,

Here, C P ∆ 1 , 0 , t Y, C u R ∆ 1 , 0 as only the mechanism for C varies across domains. Consider the evaluation of π : π p X ' 1 | c q : ' 1 {p 1 ` exp t´ c uq given an experimental dataset in M 1 in which X has been randomized, i.e. , X ' Bern p 0 . 5 q , and covariate data P 0 p C q available in M 0 . Notice that we do not have access to the specification of the SCMs M , but only the induced diagrams G ∆ , and a subset of entailed distributions P : t P 1 rand p X q p X,Y,C q , P 0 p C qu . The policy effect is expressible as

and estimated given the policy π and the combination of the available data from P 1 , P 0 .

■

X

1

W

X

2

C

1

C

2

Y

π

1

(a)

G

1

π

1

X

1

W

X

2

C

1

C

2

Y

π

2

(b)

G

2

π

2

X

1

W

X

2

C

1

C

2

Y

S

C

1

S

C

2

S

Y

π

0

π

0

(c)

G

∆

0

,

1

π

0

X

1

W

X

2

C

1

C

2

Y

S

C

1

S

C

2

S

Y

π

0

π

0

(d)

G

∆

0

,

2

π

0

3 Combining experiments from two domains

Example 1 illustrates two challenges in combining data from different domains to infer the effect of a new policy in a target domain. In a first instance highlighting the challenge of identification , that is inferring an expression in terms of P that identifies the policy effect, and in a second instance highlighting the challenge of estimation , that is providing efficient estimators from finite samples for the identified policy effect. The following example will serve to motivate this setting.

Example 2 ( Two-stage treatment strategies ) . A team of physicians is contemplating a treatment plan π 0 against heart disease Y for their patients in M 0 . They consider administrating two drugs in sequence: a drug against hypertension X 1 , followed by an anti-diabetic drug X 2 depending on the effect of X 1 on blood pressure W . To support their evaluation, two studies exist on these drugs, from domains M 1 , M 2 , that, however, have only analyzed their effect in isolation (on X 1 and X 2 separately) and under different treatment guidelines, π 1 , π 2 respectively. The data collected refers to the variables V : ' p Y, C 1 , C 2 , X 1 , X 2 , W q in which p C 1 , C 2 q are demographic variables. Formally, we assume physicians have access to P : t P 1 π 1 p V q , P 2 π 2 p V q , P 0 p C 1 , C 2 qu . The superscripts in P 0 , P 1 , P 2 are the index for the domain, and the subscripts π 0 , π 1 , π 2 denote the policies for assigning treatments. G ∆ in Fig. 1 encodes the structural assumptions, which include discrepancies across domains and implemented policies in the available data. For example, the graph G 1 π 1 specifies the known guideline π 1 used in M 1 , while no specific plan was followed for the assignment of X 2 , that in practice depends on the patient's covariates C 2 as well as unobserved factors, e.g. mood, health awareness, etc. (summarized in the bi-directed arc). In addition, selection diagrams describe differences between domains. For example, the edge t S C 1 Ñ C 1 u in G ∆ 0 , 1 π 0 indicates a potential change in the distribution of covariates C 1 across domains M 0 , M 1 . The question then becomes how to estimate E P 0 π 0 r Y s given p G ∆ , P q . ■

3.1 Identification

Example 2 illustrates the complexity of drawing inferences from multiple datasets collected under different settings. We extend this example to provide a general identification procedure for the effect of policies when two source datasets subject to different policies and/or discrepancies with the target domain are available. Let V : ' p Y, C , X 1 , W , X 2 , W , Y, S q denote a set of disjoint variables, where Y is an outcome variable, C , p C , W q are covariates corresponding to two experiments, p X 1 , X 2 q are treatment variables, and S denotes the selection nodes describing discrepancies across pairs of domains. Formally, the task signature is given as follows:

Input : Samples from P ' t P 1 π 1 p V q , P 2 π 2 p V q , P 0 p C 1 , C 2 qu ; structural assumptions G ∆ : ' t G 0 π 0 , G 1 π 1 , G 2 π 2 , G ∆ 0 , 1 π 0 , G ∆ 0 , 2 π 0 u .

Query : Estimate E P 0 π 0 r Y s where P 0 is distribution on the target domain and π 0 is a target policy assigning treatments with π 0 p X 1 | C 1 q and π 0 p X 2 | C 2 , W q .

Given these inputs, a sufficient condition for identifying the query is given as follows:

Definition 4 ( Adjustment criterion for combining two experiments ) . Given G ∆ , the adjustment criterion for combining two experimental datasets is defined by the following d-separation statements:

Domain transfer for Y : p Y K K S | C , X 1 , X 2 , W q in G ∆ 0 , 2 π 0 ; i.e., the distribution over Y is invariant between the source distribution from M 2 and the target.

Domain transfer for W : p W K K S | C , X 1 q in G ∆ 0 , 1 π 0 ; i.e., the distribution over W is invariant between the source distribution from M 1 and the target.

Adjustment for Y : p Y K K π i | C 1 , C 2 , X 1 , X 2 , W q in G π i for i P t 0 , 2 u ; i.e., the distribution over Y is invariant between regimes π 0 and π 2 .

Adjustment for W : p W K K π i | C 1 , C 2 , X 1 q in G π i for i P t 0 , 1 u ; i.e., the distribution over W is invariant between regimes π 0 and π 1 .

The adjustment criterion could be shown to hold for Example 2. Specifically, domain transfer Y could be shown by inspecting Fig.5e as the set S ' t S C 1 , S C 2 , S W u is d-separated from Y , conditional on t C 1 , C 2 , X 1 , X 2 , W u . Similarly, domain transfer for W holds as S ' t S C 1 , S C 2 , S Y u is d-separated from W , given C 1 , C 2 , X 1 in Fig.5d. Similarly, one could verify the adjustment condition for Y by inspecting Fig.5c and the adjustment condition for W by inspecting Fig. 5a.

For this example, these conditions imply identifiability of the target query E P 0 π 0 r Y s given p G ∆ , P q .

Theorem 1 ( Adjustment for combining two experiments ) . Under the adjustment criterion in Def. 4, the target query ψ 0 : ' E P 0 π 0 r Y s is identifiable from the samples from P 1 π 1 p V q , P 2 π 2 p V q , P 0 p C 1 , C 2 q . Specifically, it's expressed as follows 3 :

Effectively, despite the differences across domains encoded in Example 2, the effect of the new combination of anti-diabetic and anti-hypertensive drugs π 0 , can be estimated using samples from experiments already conducted in M 1 , M 2 , and baseline characteristics of patients in M 0 .

3.2 Estimation

This section considers the estimation of the effect of policies, building on the identification criterion in Thm. 1. We first parameterize the identification estimand in Eq. (1) with two types of nuisance parameters µ and ω . µ is a collection of regression parameters, and ω is a collection of the ratio of distributions.

The regression nuisance parameters are defined as follows: µ 2 0 p C , W, X q : ' E P 2 π 2 r Y | C , W, X s and ˇ µ 2 0 p C , W, X 1 q : ' ř x 2 µ 2 0 p C , W, X 1 , x 2 q π 0 p x 2 | C , W, X 1 q . Recursively, µ 1 0 p C , X 1 q : ' E P 1 π 1 r ˇ µ 2 0 p C , W, X 1 q | C , X 1 s and ˇ µ 1 0 p C q : ' ř x 1 µ 1 0 p C , x 1 q π 0 p x 1 | C q . Eq. (1) can be parameterized as E P 0 π 0 r Y s ' E P 0 r ˇ µ 1 0 p C qs .

On the other hand, the ratio nuisance parameters ω 2 0 , ω 1 0 are defined as functionals satisfying the following properties:

Aclosed form of the ω i 0 is provided in the later section at Eq. (14). By the definition of ratio nuisances, Eq. (1) can be parameterized as E P 0 π 0 r Y s ' E P 2 π 2 r ω 2 0 p C , W, X q Y s . Equipped with these nuisances, we now present the DML-based estimator for the target query:

Definition 5 ( DML for combining two experiments ) . Let D 2 ' P 2 π 2 p V q , D 1 ' P 1 π 1 p V q and D 0 ' P 0 p C q . Let L ě 2 denote a fixed number.

Sample split : For ℓ ' 1 , ¨ ¨ ¨ , L , randomly split D i for i P t 0 , 1 , 2 u into L -fold. The ℓ 'th partition of the sample is denoted D i ℓ . The complement is D i ´ ℓ : ' D i z D i ℓ .

Nuisance estimation : For each ℓ ' 1 , ¨ ¨ ¨ , L , learn the estimator model ˆ µ 2 ℓ and ˆ µ 1 ℓ for µ 2 0 , µ 1 0 using samples D 2 ´ ℓ , D 1 ´ ℓ , respectively. Also, learn the estimation model for ˆ ω 1 ℓ , ˆ ω 2 ℓ for ω 1 0 , ω 2 0 using samples D i ´ ℓ for i ' 0 , 1 , 2 , respectively.

Evaluation : The DML estimator ˆ ψ for E P 0 π 0 r Y s is then given as

Estimating the ratio nuisance t ˆ ω 1 , ˆ ω 2 u can be challenging due to the necessity of estimating density ratios like P 1 π 1 p C q P 2 π 2 p C q or P 1 π 1 p W | C ,X 1 q P 2 π 2 p W | C ,X 1 q . We employ the classification-based method for estimating the density [17, Sec. 5.4]. To illustrate this method, consider estimating P 1 π 1 p C q P 2 π 2 p C q . We assign λ ' 1 if samples of C are from P 1 π 1 and λ ' 0 if from P 2 π 2 . Then, it's provable that P 1 π 1 p C q P 2 π 2 p C q ' P p λ ' 1 | C q P p λ ' 0 | C q , which can be estimated using off-the-shelf probabilistic classification estimators.

The error of the DML estimator is presented below:

Theorem 2 ( Learning Guarantees ) . Suppose ˆ µ 2 ℓ , ˆ µ 1 ℓ ă 8 and 0 ă ˆ ω 2 ℓ , ˆ ω 1 ℓ ă 8 . Define ϕ 2 p V ; µ 2 , π 2 q : ' ω 2 p C , W, X qt Y ´ µ 2 p C , W, X qu , ϕ 1 pp C , W, X 1 q ; ˇ µ 2 , µ 1 , ω 1 q : ' ω 1 p C , X 1 qt ˇ µ 2 p C , W, X 1 q ´ µ 1 p C , X 1 qu , and ϕ 0 p C ; ˇ µ 1 q : ' ˇ µ 1 p C q ´ ψ 0 . For i ' 0 , 1 , 2 , define ϕ i 0 as ϕ i equipped with true nuisances ( µ i 0 , π i 0 ) and ˆ ϕ i ℓ as ϕ i equipped with estimated nuisances ˆ µ i ℓ , ˆ π i ℓ . Define R i : ' p 1 { L q ř L ℓ ' 1 p E D i ℓ r ˆ ϕ i ℓ s ´ E P i r ˆ ϕ i ℓ sq for i ' 0 , 1 , 2 . Then,

The error ˆ ψ ´ ψ 0 is decomposed as follows:

Let ρ 2 i, 0 : ' V P i π i r ϕ i 0 s . With probability (W.P) greater than 1 ´ ϵ ,

Let κ 3 i, 0 : ' E P i π i r| ϕ i 0 | 3 s . Let Φ p x q denote the standard normal CDF. W.P greater than 1 ´ ϵ ,

If the nuisance parameters ˆ µ i ℓ and ˆ π i ℓ converge at a rate of n ´ 1 { 4 (where n is the size of the smallest sample set), the DML estimator achieves a faster convergence rate of n ´ 1 { 2 . This rapid convergence allows its asymptotic distribution to closely approximate the standard normal distribution, as is further clarified in the asymptotic analysis:

Theorem 3 ( Asymptotic Error ) . Suppose each nuisance estimates ˆ µ 2 ℓ , ˆ µ 1 ℓ , ˆ ω 2 ℓ , ˆ ω 1 ℓ are L 2 -consistent and bounded. Then, the error of the DML estimator ˆ ψ in Def. 5 is given as follows:

where R i converges in distribution to normal p 0 , ρ 2 i, 0 q .

Eq. (7) implies that the error term ˆ ψ ´ ψ 0 converges to zero faster than the convergence rate of nuisances, which is a property known as debiasedness .

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

1

(a)

G

1

π

1

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

2

(b)

G

2

π

2

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

3

(c)

G

3

π

3

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

0

π

0

π

0

S

C

1

S

C

2

S

C

3

S

W

2

S

Y

(d)

G

∆

0

,

1

π

0

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

0

π

0

π

0

S

C

1

S

C

2

S

C

3

S

W

1

S

Y

(e)

G

∆

0

,

2

π

0

C

1

X

1

W

1

C

2

X

2

W

2

C

3

X

3

Y

π

0

π

0

π

0

S

C

1

S

C

2

S

C

3

S

W

1

S

W

2

(f)

G

∆

0

,

3

π

0

4 Combining multiple experiments

In this section, we extend our method to incorporate the combination of data from multiple experiments, specifically focusing on m different experiments derived from varied policies ( π i ) in distinct source domains ( M i ). A practical scenario for this task is the following:

Example 3 (Multi-stage treatment strategies) . Consider a scenario involving hospitals in three different cities: New York (domain 1 with P 1 , G 1 ), Los Angeles (domain 2 with P 2 , G 2 ), and San Francisco (domain 3 with P 3 , G 3 ). Each hospital has different guidelines, i.e., policies, for diabetes treatment. In New York, the hospital focuses on insulin therapy adjustment based on the patient lifestyle choices, primarily for Type 1 Diabetes patients ( π 1 ). In Los Angeles, the hospital focuses on team diet and exercise regimen adjustments, primarily for Type 2 Diabetes patients ( π 2 ). In contrast, San Francisco's approach involves advanced monitoring and AI-driven predictive adjustments for higher-risk diabetes patients. Now, as the leader of a new clinical team in Chicago (the target domain with P 0 , G 0 ), the task is to evaluate a novel candidate treatment policy π 0 , which integrates the strategies from these three domains to provide comprehensive care for both Type 1 and Type 2 Diabetes patients. The structure of the problem is captured in causal diagrams in Fig. 2, illustrating the data-generating process, the experiments in each city, and the assumed discrepancies between these source domains and Chicago. ■

4.1 Identification

We consider a sequence of variables p C , X 1 , W 1 , ¨ ¨ ¨ , X m , W m : ' Y q where p C , W p i ´ 1 q q represent the covariates corresponding to each of the i 'th experiments, and p X 1 , ¨ ¨ ¨ , X m q are the corresponding treatment variables. We are given samples drawn from P i π i p V q for i ' 1 , ¨ ¨ ¨ , m and P 0 p C q . We will leverage causal diagrams G π i and selection diagrams G ∆ 0 ,i for every i ' 1 , ¨ ¨ ¨ , m . Formally, the task signature is given as follows:

Input : Samples from P i π i p V q for i ' 1 , ¨ ¨ ¨ , m and P 0 p C 1 , C 2 q ; Causal diagrams G i π i and selection diagrams G ∆ 0 ,i π 0 for i ' 1 , ¨ ¨ ¨ , m .

Query : Estimate the effect of the target policy π 0 on the target domain M 0 ; i.e., E P 0 π 0 r Y s .

Definition 6 ( Adjustment criterion for combining multiple experiments ) . The adjustment criterion for combining multiple policies are the following d-separation criterion in the the DTRs G π 0 , G π 1 , ¨ ¨ ¨ , G π m and the selection diagram G ∆ 0 , 1 π 0 , ¨ ¨ ¨ , G ∆ 0 ,m π 0 .

Domain transfer for Y : p Y K K S | C , W , X q in G ∆ 0 ,m π 0 ; i.e., the distribution over Y is invariant between the source distribution from M m and the target.

Domain transfer for W i for i ' 1 , ¨ ¨ ¨ , m ´ 1 : p W i K K S | C p i q , W p i ´ 1 q q in G ∆ 0 ,i π 0 ; i.e., the distribution over W i is invariant between the source distribution from M i and the target.

Adjustment for Y : p Y K K π i | C , W , X q in G π i for i P t 0 , m u ; i.e., the distribution over Y is invariant between regimes π 0 and π m .

Adjustment for W i i ' 1 , ¨ ¨ ¨ , m ´ 1 : p W i K K π j | C p i q , X p i ´ 1 q q in G π j for j P t 0 , i u ; i.e., the distribution over W i is invariant between regimes π 0 and π i .

These conditions lead to the following identification criterion.

Theorem 4 ( Adjustment for combining multiple experiments ) . Under the adjustment criterion in Def. 6, the target query ψ 0 : ' E P 0 π 0 r Y s is identifiable from the samples from P 1 π 1 p V q , ¨ ¨ ¨ , P π m m p V q and P 0 p C q . Specifically, it's expressed as follows:

4.2 Estimation

The regression nuisance parameters are defined as follows. We first define the following nuisance.

For i ' m ´ 1 , ¨ ¨ ¨ , 1 , the other nuisances are defined in a following manner:

We note that Eq. (8) can be parameterized as E P 0 π 0 r Y s ' E P 0 r ˇ µ 1 0 p C qs . On the other hand, the ratio nuisance parameters ω i 0 for i ' 1 , ¨ ¨ ¨ , m are defined as functionals satisfying the followings:

where the closed form is given as

Eq. (8) can be parameterized as E P m πm r ω p m q p C , W p m ´ 1 q , X p m q q Y s . Equipped with these nuisances, we define a corresponding estimator as follows.

Definition 7 ( DML for combining multiple experiments ) . Let D i ' P i π i p V q for i ' 1 , ¨ ¨ ¨ , m and D 0 ' P 0 p C q . Let L ě 2 denote a fixed number.

Sample split : For ℓ ' 1 , ¨ ¨ ¨ , L , randomly split D i for i P t 0 , 1 , ¨ ¨ ¨ , m u into L -fold. The ℓ 'th partition of the sample is denoted D i ℓ . The complement is D i ´ ℓ : ' D i z D i ℓ .

Nuisance estimation : For each ℓ ' 1 , ¨ ¨ ¨ , L , learn the estimator model ˆ µ m ℓ , ¨ ¨ ¨ , ˆ µ 1 ℓ for µ m 0 , ¨ ¨ ¨ , µ 1 0 using samples D m ´ ℓ , D 1 ´ ℓ , respectively. Also, learn the estimation model for ˆ ω 1 ℓ , ¨ ¨ ¨ , ˆ ω m ℓ for ω 1 0 , ¨ ¨ ¨ , ω m 0 using samples D i ´ ℓ for i ' 0 , 1 , ¨ ¨ ¨ , m , respectively.

Evaluation : The DML estimator ˆ ψ for E P 0 π 0 r Y s is then given as

The error of the DML estimator is presented below:

Theorem 5 ( Learning Guarantees ) . Suppose µ i 0 , ˆ µ i ℓ ă 8 and 0 ă π i 0 , ˆ π i ℓ ă 8 almost surely for i ' 1 , ¨ ¨ ¨ , m . Define ϕ i pp C , W p i q , X p i q q ; ω i , ˇ µ i ` 1 , µ i q : ' ω i t ˇ µ i ` 1 ´ µ i u for i ' 1 , ¨ ¨ ¨ , m , where ˇ µ m ` 1 : ' Y . Let ϕ 0 p C ; ˇ µ 1 q : ' ˇ µ 1 ´ ψ 0 . For i ' 0 , ¨ ¨ ¨ , m , define ϕ i 0 as ϕ i equipped with true nuisances, and ˆ ϕ i ℓ as ϕ i equipped with estimated nuisances. Define R i : ' p 1 { L q ř L ℓ ' 1 p E D i ℓ r ˆ ϕ i ℓ s ´ E P i r ˆ ϕ i ℓ sq for i ' 0 , 1 , ¨ ¨ ¨ , m . Then,

The error ˆ ψ ´ ψ 0 is decomposed as follows:

Let ρ 2 i, 0 : ' V P i π i r ϕ i 0 s . With probability (W.P) greater than 1 ´ ϵ ,

3. Let κ 3 i, 0 : ' E P i π i r| ϕ i 0 | 3 s . Let Φ p x q denote the standard normal CDF. W.P greater than 1 ´ ϵ ,

A corresponding asymptotic error analysis is following:

Theorem 6 ( Asymptotic Error ) . Suppose each nuisance estimates ˆ µ 1 ℓ , ¨ ¨ ¨ , ˆ µ m ℓ and ˆ ω 1 ℓ , ¨ ¨ ¨ , ˆ ω m ℓ are L 2 -consistent and bounded. Then, the error of the DML estimator ˆ ψ in Def. 7 is given as follows:

where R i converges in distribution to Normal p 0 , ρ 2 i, 0 q .

Similarly to Thm. 3, this result implies that the DML estimator ˆ ψ converges fast even when the nuisance estimates converge relatively slowly.

5 Experiments

In this section, we demonstrate the proposed estimators in Defs. (5,7) for combining multiple experimental datasets from different domains. We first compared the estimators on synthetic data to provide evidence of the fast convergence and doubly robustness behaviours of the proposed estimators. We conclude with an analysis of the ACTG 175 clinical trial [21] and Project STAR. We will use T est p x q for est P t reg , pw , dml u to denote the estimators t OM , PW , DML u for the policy effect E P 0 π 0 Y . OM and PW estimators are purely based on the regression-based nuisances µ and ω , respectively. To assess the quality of each estimator, we consider the absolute error (AE) as AE est ' | T est p x q ´ E P 0 π 0 r Y s | . We used XGBoost [12] to estimate nuisances.

Synthetic Simulations We ran 100 simulations for each N ' t 2500 , 5000 , 10000 , 20000 u where N is the sample size. We measure the AE est in the presence of the 'converging noise ϵ ' in estimating the nuisance, decaying at a N ´ 1 { 4 rate (i.e., ϵ ' normal p N ´ 1 { 4 , N ´ 1 { 4 q , where N is the size of samples). To enforce the convergence rate of nuisance estimates no faster than the decaying rate n ´ 1 { 4 , we add ϵ to all nuisance estimates. This scenario is inspired by the experimental design discussed in [27]. The AE plots for combining two/multiple experiments are presented in Figs. (3a, 3b). For all examples, the proposed DML estimator outperforms the other two estimators by achieving fast convergence. This result corroborates the robustness property in Thm. (3, 6), which implies that the proposed estimator converges faster than the other counterparts.

(a)

(b)

(c)

(d)

0.7

0.20

OM

OM

0.6

OM

OM

0.15

PW

DML

0.25

0.20

PW

DML

0.5

0.4

PW

DML

0.6

0.5

PW

DML

A

E

0.10

E

0.15

E

0.3

0.2

E

0.4

0.05

0.10

0.1

0.3

0.00

0.05

0.0

0.2

5000

10000

20000

5000

10000

20000

500

1000

2000

500

1000

2000

Sample size (n)

Sample size (n)

Sample size (n)

Sample size (n)

External validity: ACTG 175 To provide empirical evidence, we analyze the ACTG 175 randomized trial [21], which assessed therapies for reducing CD4 T cell counts in HIV patients. Participants were randomly assigned to treatments X 2 P t 0 , 1 u , with prior anti-retroviral drug use X 1 P t 0 , 1 u recorded. Patient demographics C 1 , C 2 -including gender, age, weight, and Karnofsky score-were collected, and CD4 T cell counts ( W ) were measured. To simulate an alternative study with a modified guideline for X 1 , we sub-sampled ACTG 175, adjusting covariate distributions and assignments of t X 1 , X 2 u . Specifically, we evaluate a stochastic policy π 0 ' t π 0 p x 1 | c 1 q , π 0 p x 1 | c 2 qu for combining X 1 and X 2 based on C 1 , C 2 , with distribution P 0 representing a location with differing covariate distributions and treatment assignments. Further details are provided in Appendix D.2.

We evaluated the AE est of all proposed estimators with and without noise (as described in the synthetic simulations). The AE plots are shown in Figs. (3c, 3d). Results indicate that both the regression and DML estimators converge to the true policy effect faster under noisy conditions, whereas the PW estimator converges more slowly. However, DML does not consistently outperform at all sample sizes (see Fig. 3c), as its error is influenced by the combined errors in the OM and PW estimators. Consequently, high error in the PW estimator may lead to increased error in the DML estimator.

External validity: Project STAR We further examine policies on teacher-student ratios (i.e., class sizes) to improve academic achievement, using a semi-synthetic adaptation of the Project STAR dataset [40]. This longitudinal study evaluated the impact of teacher-student ratios on academic outcomes for students in kindergarten through third grade, with students randomized each year to one of three class size interventions. Here, we assess a 3-stage policy setting student-teacher ratios across Grades 0, 1, and 2, observing academic scores as intermediate outcomes, with baseline covariates (e.g., ethnicity, gender) and final academic scores at the end of Grade 3 as the primary outcome. To

0.5

OM

PW

0.4

DML

E

0.3

0.2

0.1

0.0

2000

5000

10000

Sample size (n)

emulate data collected across different domains, we subsample using various probabilities to shift baseline covariate distributions, as done in ACTG 175 (see Appendix D.3 for details). We evaluated the PW, OM, and DML estimators across dataset sizes, plotting their absolute errors against the true effect of the candidate policy. Results, shown in Fig. 4, mirror earlier experiments, with all estimators improving as sample size increases and DML showing faster convergence.

6 Conclusion

This paper has considered the evaluation of the effectiveness of policies in settings where the available data is sampled from distributions that differ from the population in the target domain. We have illustrated this task with the problem of extrapolating the results of a clinical trial in both working examples and real-world scenarios to evaluate variations of the treatment in different populations. Our contributions are (1) introducing several identification criteria for the effectiveness of policies given experimental datasets from two or more domains and (2) developing doubly robust estimators for these settings that achieve fast convergence.

Acknowledgments

We thank anonymous reviewers for constructive comments to improve the manuscript.

References

Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 687-696, 2017.

Elias Bareinboim, Sanghack Lee, Vasant Honavar, and Judea Pearl. Transportability from multiple environments with limited experiments. Advances in Neural Information Processing Systems , 26, 2013.

Elias Bareinboim and Judea Pearl. Transportability from multiple environments with limited experiments: Completeness results. Advances in neural information processing systems , 27, 2014.

Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences , 113(27):7345-7352, 2016.

Alexis Bellot, Alan Malek, and Silvia Chiappa. Transportability for bandits with data from different environments. Advances in Neural Information Processing Systems , 36, 2024.

Andrew Bennett and Nathan Kallus. Efficient policy learning from surrogate-loss classification reductions. In International conference on machine learning , pages 788-798. PMLR, 2020.

Andrew C Berry. The accuracy of the gaussian approximation to the sum of independent variates. Transactions of the american mathematical society , 49(1):122-136, 1941.

Rohit Bhattacharya, Razieh Nabi, and Ilya Shpitser. Semiparametric inference for causal effects in graphical models with hidden variables. The Journal of Machine Learning Research , 23(1):13325-13400, 2022.

Peter J Bickel, Chris AJ Klaassen, Peter J Bickel, Ya'acov Ritov, J Klaassen, Jon A Wellner, and YA'Acov Ritov. Efficient and adaptive estimation for semiparametric models , volume 4. Springer, 1993.

Donald T Campbell. Factors relevant to the validity of experiments in social settings. Sociological methods , pages 243-263, 2017.

Donald T Campbell and Julian C Stanley. Experimental and quasi-experimental designs for research . Ravenio books, 2015.

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 2016.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.

Thomas D Cook and Donald T Campbell. Causal inference and the language of experimentation. Quasi-experimentation: Design & analysis issues for field settings , pages 1-36, 1979.

Juan Correa and Elias Bareinboim. General transportability of soft interventions: Completeness results. Advances in Neural Information Processing Systems , 33:10902-10912, 2020.

Juan D Correa and Elias Bareinboim. From statistical transportability to estimating the effect of stochastic interventions. In IJCAI , pages 1661-1667, 2019.

Iván Díaz, Nicholas Williams, Katherine L Hoffman, and Edward J Schenck. Nonparametric causal effects based on longitudinal modified treatment policies. Journal of the American Statistical Association , 118(542):846-857, 2023.

Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.

Carl-Gustav Esseen. On the liapunov limit error in the theory of probability. Ark. Mat. Astr. Fys. , 28:1-19, 1942.

Stuart S Glennan. Mechanisms and the nature of causation. Erkenntnis , 44(1):49-71, 1996.

Scott M Hammer, David A Katzenstein, Michael D Hughes, Holly Gundacker, Robert T Schooley, Richard H Haubrich, W Keith Henry, Michael M Lederman, John P Phair, Manette Niu, et al. A trial comparing nucleoside monotherapy with combination therapy in hiv-infected adults with cd4 cell counts from 200 to 500 per cubic millimeter. New England Journal of Medicine , 335(15):1081-1090, 1996.

Tobias Hatt, Daniel Tschernutter, and Stefan Feuerriegel. Generalizing off-policy learning under sample selection bias. In Uncertainty in Artificial Intelligence , pages 769-779. PMLR, 2022.

Li He, Long Xia, Wei Zeng, Zhi-Ming Ma, Yihong Zhao, and Dawei Yin. Off-policy learning for multiple loggers. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1184-1193, 2019.

Yonghan Jung, Iván Díaz, Jin Tian, and Elias Bareinboim. Estimating causal effects identifiable from a combination of observations and experiments. Advances in Neural Information Processing Systems , 36, 2024.

Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects on markov equivalence class through double machine learning. In International Conference on Machine Learning , pages 5168-5179. PMLR, 2021.

Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating joint treatment effects by combining multiple experiments. In International Conference on Machine Learning , pages 15451-15527. PMLR, 2023.

Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics , 17(2):3008-3049, 2023.

Edward H Kennedy, Sivaraman Balakrishnan, Max G'Sell, et al. Sharp instruments for classifying compliers and generalizing causal effects. Annals of Statistics , 48(4):2008-2030, 2020.

Sanghack Lee, Juan Correa, and Elias Bareinboim. General transportability-synthesizing observations and experiments from heterogeneous domains. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10210-10217, 2020.

Sanghack Lee, Juan D Correa, and Elias Bareinboim. Generalized transportability: Synthesis of experiments from heterogeneous domains. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , 2020.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review. and Perspectives on Open Problems , 5, 2020.

Peter Machamer, Lindley Darden, and Carl F Craver. Thinking about mechanisms. Philosophy of science , 67(1):1-25, 2000.

Susan A Murphy. An experimental design for the development of adaptive treatment strategies. Statistics in medicine , 24(10):1455-1481, 2005.

Sonali Parbhoo, Shalmali Joshi, and Finale Doshi-Velez. Generalizing off-policy evaluation from a causal perspective for sequential decision-making. arXiv preprint arXiv:2201.08262 , 2022.

Judea Pearl. Causality . Cambridge university press, 2009.

Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. In Twenty-fifth AAAI conference on artificial intelligence , 2011.

Andrea Rotnitzky, James Robins, and Lucia Babino. On the multiply robust estimation of the mean of the g-functional. arXiv preprint arXiv:1705.08582 , 2017.

IG Shevtsova. On the absolute constants in the berry-esseen-type inequalities. In Doklady Mathematics , volume 89, pages 378-381. Springer, 2014.

Chengchun Shi, Jin Zhu, Ye Shen, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association , 119(545):273-284, 2024.

James H Stock and Mark W Watson. Introduction to econometrics . Pearson, 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems , 28, 2015.

Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10276-10283, 2020.

Masatoshi Uehara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under a covariate shift. Advances in Neural Information Processing Systems , 33:49-61, 2020.

Mark J van der Laan and Susan Gruber. Targeted minimum loss based estimation of causal effects of multiple time point interventions. The international journal of biostatistics , 8(1), 2012.

Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandit: a causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 1778-1780, 2017.

Supplement to 'Efficient Policy Evaluation Across Multiple Different Experimental Datasets'

Contents

A Related Work

Evaluating the impact of a policy using observational and experimental data under different conditions is a widespread challenge in various important decision-making fields. Formulations of this problem have appeared in the causality literature, but also in statistics, reinforcement learning, and epidemiology.

Off-policy evaluation (OPE) aims to assess the performance of policies of interest using observational samples. In this line of research, [44] considers generalizing the effect of a policy under distribution shift. We build on this intuition, but instead seek to combine multiple (policy-)interventional data from source domains to learn the effect of policies of interest on the target domain. [1, 23] for instance used auxiliary datasets from multiple bandit instance, though they setting assume that the datasets are sampled from the same underlying populations and environments. Several authors have also considered transfer learning in off-policy learning in the context of bandits [46, 5]. Further, [22] addresses the problem of selection biases in observational data for off-policy learning.

In the causal inference literature, combining multiple experimental studies to estimate a new causal effect is a task called generalized identification [30]. Recent progress has been made in developing corresponding estimators [26, 24]. However, these estimators are not applicable when our goal is to combine multiple policy interventional studies from source domains to estimate a causal effect in the target domain. Accordingly, [4, 29, 15] developed the notion of generalized transportability that aims to evaluate a causal effect on a target domain from multiple observational and / or interventional distributions from other source domains. In this line of research, our work relates closely to Correa and Bareinboim's identification algorithm for the effect of policies [15]. We similarly develop identification criteria that are conducive to efficient estimation from finite samples. In particular, our work focuses on the derivation of sample-efficient estimators for the policy effect of interest on the target domain.

From this perspective, our work can be interpreted as a bridge between causal inference and off-policy evaluation [34] since we leverage formal theories in causal inference (e.g., generalized identification [29], generalized transportability [15, 30]) to solve off-policy evaluation problems efficiently from finite samples. There are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature use the backdoor adjustment to learn the Q value as a function of the state, to address confounding effects [41]. Meanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders [39]. Finally, some works have leveraged double negative controls for OPE [43].

B Broader Impact Statement

Our work investigates the conditions under which policies may be estimated from multipe datasets collected under different conditions. In this work, we start from the assumption that causal and selection diagrams that are consistent with the underlying data generating systems of interest are available. In general, this requires domain knowledge and should be justified by prior knowledge or experiment. It is important also to make the distinction between the task of partial identification, that is inferring an expression for bounds on causal effects, and that of estimation, that is providing efficient estimators from finite samples to compute bounds in practice. This set of results concerns mostly the second task. In higher-dimensional systems, the computational complexity of estimating the conditional expectations and density ratios that define our estimators could be a substantial challenge. Consequently, practitioners must exercise caution when deploying the proposed method in small sample scenarios where estimators may be inaccurate. Moreover, we have stated our convergence guarantees in the infinite sample limit, without quantifying the finite-sample estimation uncertainty. Finally, we emphasize that simulations on real and synthetic data are provided for illustration purposes only. These results do not recommend or advocate for the implementation of a particular policy, and should be considered in practice in combination with other aspects of the decision-making process.

C Proofs

C.1 Proof for Theorem 1 and Theorem 4

Since Theorem 1 is a special case for Theorem 4, we will only prove for Theorem 4.

Note

Then,

by leveraging the domain transfer for Y and adjustment for Y condition.

For each P 0 π 0 p w i | c , X p i ´ 1 q , w p i ´ 1 q q ,

again, by leveraging the domain transfer condition for W i and adjustment condition for W i . This completes the proof.

C.2 Proof for Theorem 2 and Theorem 5

Since Theorem 2 is a special case for Theorem 5, we will only prove for Theorem 5. Throughout the proof, we will use C 1 : ' C , X i : ' t X i u for i ' 1 , ¨ ¨ ¨ , m , and C i : ' t W i ´ 1 u for i ' 2 , ¨ ¨ ¨ , m ´ 1 . Also, we will sometimes use P 1 p C 1 q : ' P 0 p C q , P i p C i | C p i ´ 1 q Y X p i ´ 1 q q for i ą 1 as P i ´ 1 π i ´ 1 p W i ´ 1 | C , W p i ´ 2 q , X p i ´ 1 q q .

C.2.1 Proof of Mixed Bias Property

Using the fact that ψ 0 ' E P 0 r ˇ µ 1 0 s , we can write it as

Then, we will claim and prove the following:

Lemma 1 ( Mixed Bias Property ) . Suppose µ i 0 , ˆ µ i ă 8 and 0 ă π i 0 , ˆ π i ă 8 almost surely for i ' 1 , ¨ ¨ ¨ , m . For i ' 1 , 2 , ¨ ¨ ¨ , m , define

and ˇ µ m ` 1 : ' Y . Define ϕ 0 p C ; ˇ µ 1 q : ' ˇ µ 1 . For i ' 0 , ¨ ¨ ¨ , m , define ϕ i 0 as ϕ i equipped with true nuisances, and ˆ ϕ i as ϕ i equipped with estimated nuisances. Then,

Proof of Lemma 1 . For i ' m, ¨ ¨ ¨ , 1 with ˇ µ m ` 1 : ' Y , define

Then,

For i ' m ´ 1 , ¨ ¨ ¨ , 1 ,

Also, the following holds:

Finally, E P 1 π 1 r ω 1 0 ˆ µ 1 s ' E P 1 π 1 r ˇ µ 1 s .

Therefore,

C.2.2 Proof for Statement 1

Recall

From Eq. (7),

Then, the error ˆ ψ ´ ψ 0 can be decomposed into

Define

Then, the error can be represented as

C.2.3 Proof for Statement 2

We will use the following results:

Lemma2 ( Combining concentration inequalities ) . Suppose P p A k ą t q ď b k { t 2 for k ' 1 , ¨ ¨ ¨ , K . Then,

Proof of Lemma 2 . The event ř K k ' 1 A k ď tK includes the case where A k ă t for k ' 1 , ¨ ¨ ¨ , K . Therefore,

Lemma 3 ( Stochastic Equicontinuity ) . Let D iid ' P . Let D ' D 0 ¨ Y D 1 , where n : ' | D 0 | . Let ˆ f be a function estimated from D 1 . Then, in probability greater than 1 ´ ϵ for any ϵ P p 0 , 1 q ,

Proof of Lemma 3 . This proof is from [28, Lemma 2]. Since ˆ f is a function of D 1 , we will denote ˆ f D 1 . Define a following random variable of interest:

Then, the conditional expectation of X given D 1 is zero, since

where the third equality holds by the independence of D 0 and D 1 . Therefore,

which implies that

Also,

By applying the (conditional-) Chevyshev's inequality,

Then,

In other words, X ă t in probability greater than 1 ´ 1 nt 2 } ˆ f D 1 ´ f } 2 P . If t ' } ˆ f D 1 ´ f } P ? nϵ , then X ă } ˆ f D 1 ´ f } P ? nϵ in the probability greater than 1 ´ ϵ for any ϵ P p 0 , 1 q .

Here, we will study the finite sample behavior of

By Chevyshev's inequality, or equivalently,

Choose t 1 : ' c 2 ϵ ř m i ' 0 ρ 2 i, 0 | D i | and t 2 : ' d 2 ϵ ř L ℓ ' 1 ř m i ' 0 } ˆ ϕ i ℓ ´ ϕ i 0 } 2 P i π i | D i ℓ | . Then, with a probability greater than 1 ϵ ,

C.2.4 Proof for Statement 3

We will use the following result:

Proposition 1 ( Berry-Esseen's inequality [7, 19, 38]) . Suppose D ' t X 1 , ¨ ¨ ¨ , X n u are independent and identically distributed random variables with E P r X i s ' 0 , E P r X 2 i s ' σ 2 and E P r| X i | 3 s ' κ 3 . Then, for all x and n ,

With a probability greater than 1 ´ ϵ ,

Define

Define

Then, R i ' A i ` B i . Then,

Then,

This completes the proof.

C.3 Proof for Theorem 3 and Theorem 6

By Cauchy-Schwartz' inequality,

Given assumption, the upper bound in Eq. (18) converges at 1 { a | D i ℓ | rate. Therefore, R i converges in distribution to normal p 0 , ρ 2 i, 0 q .

D Details of Simulations

D.1 Data Generating Process for Synthetic Simulations

Codes corresponding to simulations are submitted as supplementary materials.

D.1.1 Synthetic Simulations for Fig. 3a

Wedefine the following SCM. First, U XW , U X 1 ,X 2 , U X 2 ,W , U X 2 ,Y , U C 1 , 1 , U C 1 , 2 , U C 2 , 1 , U C 2 , 2 , U W , U Y ' normal p 0 , 1 q . Then,

■

Also, for S ‰ 0 ,

D.1.2 Synthetic Simulations for Fig. 3b

We define the following SCM. First, U X 1 ,X 2 , U X 2 ,X 3 , U W 1 ,X 1 , U W 1 ,X 2 , U W 2 ,X 2 , U W 2 ,X 3 , U C 1 ,C 2 , U C 2 ,C 3 , U X 3 ,Y , U C 1 , 1 , U C 1 , 2 , U C 2 , 1 , U C 2 , 2 , U C 3 , 1 , U C 3 , 2 , U W 1 , U W 2 , U Y ' normal p 0 , 1 q . Then,

Also, for S ‰ 0 ,

D.2 External validity of the ACTG 175 clinical trial

To provide empirical evidence of policy estimation in a real-world setting, we revisit the ACTG 175 randomized clinical trial from 1994 conducted on patients from the United States and Puerto Rico [21]. It investigated the effectiveness of different therapies for reducing CD4 T cell counts in individuals with HIV (selected subject to various inclusion criteria). In the study, individuals were randomly assigned to two different treatments X 2 P t 0 , 1 u , and a record was made on whether a previous anti-retroviral drug had been administered X 1 P t 0 , 1 u prior to the start of the trial. Patient demographics C 1 , C 2 including gender, age, weight, among others, were collected, and CD4 T cell count were measured at treatment time W , and again 20 weeks after treatment initialization Y , the outcome of the analysis. To simulate a second study with a different guideline for anti-retroviral drug administration, we considered a sub-sampled version of ACTG 175 in which covariate distributions as well as the assignment of X 1 , X 2 were modified.

ACTG 175 is an experimental study in which X 2 has been randomized and X 1 follows a baseline, unknown, stochastic policy π 2 : Ω C 2 ˆ Ω X 1 Ñ r 0 , 1 s assumed to depend on study-specific features C 2 such a patient's Karnofsky score and symptomatic indicators (both normalized to lie in the r 0 , 1 s interval). Samples of variables C 1 , C 2 , W, X 1 , X 2 , Y therefore follow a distribution P 2 rand p X 2 q ,π 2 p C 1 , C 2 , W, X 1 , X 2 , Y q . The suffix 'rand p X 2 q ' denotes a policy that randomizes X 2 , i.e. X 2 ' Bern p 0 . 5 q .

X

1

W

X

2

C

1

C

2

Y

π

2

(a)

G

ACTG

'

G

2

π

2

X

1

W

X

2

C

1

C

2

Y

π

1

(b)

G

1

π

1

X

1

W

X

2

C

1

C

2

Y

(c)

G

0

X

1

W

X

2

C

1

C

2

Y

S

C

1

S

C

2

S

Y

π

0

π

0

(d)

G

∆

0

,

1

π

0

X

1

W

X

2

C

1

C

2

Y

S

C

1

S

C

2

S

Y

π

0

π

0

(e)

G

∆

0

,

2

π

0

We generate a data sample from a second domain ( S ' 1 ) following the marginalized distribution P 1 π 1 p C 1 , W, X 1 q , mimicking a simple stochastic guideline on X 1 in which π 1 : ' π 1 p x 1 ' 1 | c 1 q ' 1 {p 1 ` exp t´ c 11 ´ c 12 ´ c 13 uq . Higher values of C 1 (taken to be normalized measurements of weight, height, and age) lead to higher likelihood of treatment. We achieve this dataset by sampling according to a re-weighted version of the ACTG 175 trial. In particular, we collect data from P 1 according to,

For this example, we consider evaluating a stochastic policy π 0 ' t π 0 p x 1 | c 1 q , π 0 p x 1 | c 2 qu that combines the drugs X 1 , X 2 according to a stochastic policy for X 1 based on weight, height, and age, p C 1 q and for X 2 based on a patient's Karnofsky score and symptomatic indicators p C 2 q . In particular,

The policy π 0 is considered to be implemented on a patient population located in a different location that are know to have a differing covariate distribution P 0 p C 1 , C 2 q to that observed in ACTG 175 and the second study, among other discrepancies. We assume that the SCM generating this experimental study follows the causal graphs in Fig. 5.

The target population under π 0 is then given by

We limit all datasets to approximately 2000 samples as this is the size of the ACTG 175 trial. The ground truth target effect E P 0 π 0 r Y s is evaluated by taking the empirical mean of Y in the sample of data collected from P 0 with the procedure above.

D.3 External validity of the Project STAR study

We describe in this section additional experimental details on the Project STAR study 4 . This study investigated the impact of teacher/student ratios on academic achievement for kindergarten through third-grade students. Project STAR was a four-year longitudinal study where students were randomly assigned to one of three interventions with different class sizes each year, following different randomization procedures. The causal diagram we assume for this setting is provided in

Fig. 6. Bi-directed arcs denote unobserved confounding in the observational regime (when student are observed in a particular class size rather than forced to join a particular class size).

Specifically, we consider the evaluation of a 3-stage stochastic policy,

where,

These policies determine the student-teacher ratio X 0 , X 1 , X 2 , taking values "regular" or "small", across three different grades, namely Grade 0 (Kindergarten), Grade 1 and Grade 2. C refers to a two-dimensional demographic variable encoding gender and ethnicity, converted to binary and categorical variables respectively. (To avoid clutter, in Fig. 6 we use C ' C 1 ' C 2 ' C 3 .) W 1 , W 2 are intermediate school outcomes that include the sum total of an individual's reading score and math score in grades 0 (Kindergarten) and 1 respectively. Y is the outcome of interest and represents total reading score and math score in grade 2.

To mimic the setting where data at different stages was collected from different domains, we subsample the dataset using different sets of probabilities to induce differences in the distributions of baseline covariates. In particular, we fix the dataset in the target domain ( S ' 0 ) to the distribution observed in the study and sub-sample according to different probabilities to create datasets for domains S ' 1 , S ' 2 , and S ' 3 , as follows.

We generate a sample of data from a first source domain ( S ' 1 ) following the marginalized distribution P 1 π 1 p C 1 , W 1 , X 1 q , where π 1 : ' π 1 p x 1 ' 1 | c 1 q ' 1 {p 1 ` exp t 0 . 5 p c 11 ` c 12 q ´ 1 uq defines the probability for the student-teacher ratio variables in Kindergarten in domain S ' 1 . We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 1 according to,

where P 1 p c 1 q ' 0 . 3 if c 11 ' 1 , c 12 ' 1 and P 1 p c 1 q ' 0 . 7 otherwise.

We generate a sample of data from a second source domain ( S ' 2 ) following the marginalized distribution P 2 π 2 p C 2 , W 1 , W 2 , X 1 , X 2 q , where π 2 : ' π 2 p x 2 ' 1 | c 2 , x 1 , w 1 q ' 1 {p 1 ` exp t 0 . 5 p c 21 ` c 22 q ` 2 x 1 ´ 1 ´ 0 . 5 w 1 ´ 1 uq defines the probability for the student-teacher ratio variables in grade 1 in domain S ' 2 . We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 2 according to,

where P 2 p c 2 q ' 0 . 7 if c 21 ' 1 , c 22 ' 1 and P 2 p c 2 q ' 0 . 3 otherwise.

We generate a sample of data from a third source domain ( S ' 3 ) following the marginalized distribution P 3 π 3 p C 3 , W 1 , W 2 , X 1 , X 2 , X 3 q , where π 3 : ' π 3 p x 3 ' 1 | c 3 , x 1 , w 1 , x 2 , w 2 q ' 1 {p 1 ` exp t 0 . 5 p c 31 ` c 32 q ` 2 p 2 x 1 ´ 1 q ´ 0 . 25 w 1 ` p 2 x 2 ´ 1 q ` 0 . 25 w 2 ´ 1 uq defines the probability for the student-teacher ratio variables in grade 3 in domain S ' 3 . We achieve this dataset by sampling according to a re-weighted version of the STAR study. In particular, we collect data from P 3 according to,

where P 3 p c 3 q ' 0 . 5 if c 31 ' 1 , c 32 ' 1 and P 3 p c 3 q ' 0 . 5 otherwise.

X

1

W

1

X

2

C

1

C

2

C

3

W

2

X

3

Y

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made match the theoretical and experimental results presented in the paper. A broader overview statement in the Appendix reflects how much the results can be expected to generalize to other settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a broader overview statement in the Appendix to more thoroughly describe the limitations of our analysis, assumptions, and applicability in real-world settings.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical statement are quoted in full in the paper. We have attempted to provide an example to illustrate the significance of each theoretical statement and highlight its implications. The formal proof of all statements is given in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the data generating mechanisms, details of the target of estimation and information as to what python libraries can be used to fit the proposed estimators. We do not, however, disclose an open source implementation of the proposed methods at this moment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data is publicly available and we have provided full details as to where to access the data and how to run the synthetic data generation pipeline. The code will not be open sourced at this moment but we believe to have provided sufficient details to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer:

[TODO]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provided details where applicable. In our case, data splits, hyperparameters, optimizers, etc., are not significant for the implementation the method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report our results with error bars that represent 2 standard deviations from the mean.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the guidelines and we do not think that our work presents any notable concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include a broader overview statement in the Appendix. We do not expect any negative societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing asset

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: The paper does not involve crowd-sourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies how to evaluate policies where source and target sites have distribution shifts. The authors introduce identification criteria for the effectiveness of policies, and develop doubly robust estimators that achieves fast convergence. The results are also generalized to multiple source datasets. Simulation results are provided to show the effectiveness of the method.

STRENGTHS:
1. This paper is well written and easy to follow. The setups are clearly introduced and motivated, the assumptions and methodologies are accurately stated. 

2. The proposed framework is general to cover both two domains and multiple domains.

3. The empirical results seem good and well aligned with the theories for both synthetic simulation and real world datasets.

WEAKNESSES:
I am not an expert in this field and not familiar with causal inference. I find no major weaknesses. Some minor weaknesses are on the empirical evaluations. For example, the non-synthetic experiments are only conducted on ACTG 175 clinical trial dataset. Experimenting on other different datasets will enhance the empirical results.

QUESTIONS:
See weaknesses part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops novel graphical criteria and estimators for evaluating the effectiveness of various policies by integrating data from multiple experimental studies.  Through theoretical analysis and empirical validation via simulations, the paper demonstrates the robustness and fast convergence of the proposed estimators.\n\nSTRENGTHS:\nThe authors develop nonparametric identification criteria to determine whether the effect of a policy may be expressed through an adjustment formula from two separate distributions induced by policy interventions collected from different populations. Further, they generalize the identification criteria and propose a general multiplier robust estimator applicable to the evaluation of policies from multiple source datasets.\n\nWEAKNESSES:\nThe paper is difficult to follow. It is very hard to understand. Some grammatical mistakes and confused notations, e.g.,\n1. line 71, ‘variable $\\mathbf{Z}$ For example’ -> ‘variable $\\mathbf{Z}$. For example’\n2. unify the notations if necessary, try to avoid writing $\\bigtriangleup_{ij}$ and $\\bigtriangleup_{i, j}$ simultaneously.\nFor the experimental section, please give detailed descriptions on the simulated experiment and the empirical experiment (if the descriptions are too long, please move to the appendix), which allows reader to understand the advantages of the proposed framework.\nFor the theoretical section, the authors first give study when combining two experiments, followed by the study when combining multiple experiments. Honestly, the materials in the two studies are similar. Authors can unify the two studies: presenting the study when combining multiple experiments followed by the two examples (example 2 and example 3). This can release more space to discuss the core components.\n\nQUESTIONS:\nConsider example 3. The proposed framework allows researchers to evaluate a policy from the three experiments. Suppose that all the datasets are synthetic together such that the resulting data is treated as a single dataset obtained from one experiment; probably, we can evaluate a policy based on the synthetic dataset as well. Could you conduct a study that compares the two policy evaluation methods?\n\nIn some situations, it is necessary to evaluate a policy when the size of the multiple treatments is large. Consider example 3 again, but this time, we consider all the big cities in the US instead of three cities in the US. The computation complexity increases, obviously. Are there any efficient methods to handle the relevant problem?\n\nThe adjustment criterion assumes that the distribution is invariant across various experiments. To my realization, the distributions vary even in the examples presented in the paper. How does the proposed methodology handle situations where the underlying assumptions are violated?\n\nLIMITATIONS:\nThe paper focuses on combining two experiments for policy evaluation only. It would be better if the authors can discuss the optimal policy when combining two experiments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies off-policy evaluation in a transfer learning setting with multiple source datasets collected from observational and/or randomized studies. The objective is to evaluate the effect of a target policy on a possibly different target population. To achieve this, the author(s) assume at each time point, at least one source population shares the same distribution of the dynamic variable given the historical covariate-treatment pair as the target population. This ensures the identifiability of the target population's policy value, even no source population perfectly matches its entire distribution over time. The author(s) further propose doubly robust estimators, investigate their theoretical properties and finite sample performance.\n\nSTRENGTHS:\n**Originality**: To the best of my knowledge, the proposed identification formula and the proposed estimators introduced have not yet appeared in the existing literature.\n\n**Quality**: The theorems established seem to be correct, and the methodologies proposed are theoretically sound and potentially useful.\n\n**Clarity**. Overall, the paper is well-organized and easy to follow.\n\nWEAKNESSES:\nI have two major concerns, regarding the technical assumptions and numerical experiments, along with several moderate comments. I detail them one by one below:\n\n**Assumptions**. The assumptions might appear overly realistic:\n\n* **Same number of studies to horizon**. In addressing a $T$-horizon off-policy evaluation (OPE) problem, the author(s) assume that there are exactly $T$ studies, each corresponding to a time point $t$, where exactly one study per time point shares the same outcome distribution (the distribution of $Y_T$ if $t=T$ and that of $W_{t+1}$ otherwise) with the target population at that time. These settings seem unrealistic in practical scenarios. It would be beneficial for the author(s) to provide examples of real applications that validate this assumption. The current analysis using the ACTG study seems artificial; I will elaborate on this concern in more detail later.\n* **Knowledge of matching source population**. Additionally, the author(s) seem to require prior knowledge of which source population matches the target population’s outcome distribution at each time point. In practical scenarios, while we may have access to multiple studies, it is typically unknown which one aligns with the target's outcome distribution at each time point. A more realistic approach would adaptively learn which studies are similar to the target distribution at each time based on the data. This adaptive learning scenario, in my opinion, would better fit real-world applications.\n* **Identical distributions**. Although the outcome distributions between the source and target populations may be similar at each time, they are not necessarily identical. Even if the distributions are not exactly the same, as long as the differences are minimal, it is reasonable to use source data for transfer learning remains. Recent studies have addressed such distributional shifts using regularization or adaptive weighting: \n    - https://arxiv.org/pdf/2112.09313\n    - https://arxiv.org/pdf/2111.15012\n    - https://arxiv.org/pdf/2406.00317\n\n**Numerical experiments**. The experiments are overly simplified:\n\n* **Single-horizon settings.** While the paper studies multi-horizon dynamic treatment regimes, the simulations are conducted in a single stage setting. I would suggest to use D4RL benchmark datasets or OpenAI Gym environments to evaluate the proposed methodologies in multi-stage studies.\n* **Lack of competing methods.** There are some naive estimator to consider in these transfer learning settings. However, the author(s) did not include them in the experiments. For instance, one can ignore the differences in the outcome distributions and assume the multiple datasets come from a single population, with a mixture of multiple behavior policies that generates the action. \n* **Real data**. The ACTG dataset is generated under a single population. To evaluate the proposed methodology, the author(s) manually created a second population and a target population. It would be better if a real \"multi-source\" dataset could be used.\n\n**Related works**. \n* Under the identical distribution assumption, this paper mainly considers settings with covariate shift. There are some recent works that studied similar problems, although in single-stage settings, e.g.,  https://onlinelibrary.wiley.com/doi/epdf/10.1111/biom.13583. \n* In the related work section, the author(s) argued that their work can be viewed as a bridge between causal inference and OPE by  leveraging formal theories in causal inference to solve OPE problems. In that sense, there are prior works that similarly integrated both fields. For instance, standard policy evaluation methods in the RL literature uses the backdoor adjustment to learn the Q- or value as a function of the state, to address the confounding effects (see e.g., https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf). Meanwhile, other studies have applied the front-door adjustment formula for OPE in the presence of unmeasured confounders (https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2110878). Finally, some works have leveraged double negative controls for OPE (https://cdn.aaai.org/ojs/6590/6590-13-9815-1-10-20200520.pdf). \n\n**Typo**: $Z$ is used to denote the dynamic covariate in Section 2. However, this notation has been replaced with $W$ starting from Section 3.\n\nQUESTIONS:\nWould it be possible to establish the semiparametric efficiency of the proposed estimators?\n\nLIMITATIONS:\nThe limitations have been discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This work presents a method for evaluating effectiveness of policies across multiple domains using a new graphical criteria and estimators  by combining data from multiple experimental studies. The authors report error analysis of the proposed estimators that gives provides fast convergence guarantee, and additionally share simulation results as well as empirical verification on real data.

STRENGTHS:
The targeted problem seem to be of importance, and the proposed method provides theoretical error and convergence analysis, backed by empirical results both based on simulation and real data.

WEAKNESSES:
The the proposed method seems to have important potentials in real world applications, though the majority of the paper has been dedicated to theoretical analysis and the empirical evaluations are very limited.
There is little discussion on how this could impact realworld problems, and there is not a detailed analysis on the empirical results of the results on ACTG175 dataset.

From figure 3.c, it can be seen a contradictory message that the proposed DML method does not perform better than OM on all datasets, though there is no discussion on that. 

also authors state that "simulations on real and synthetic data are provided for illustration purposes only." which brings the question to what extend this approach is useful in real world scenarios.

QUESTIONS:
- Please provide more empirical analysis on the proposed method ideally on more real-world datasets if possible.

- Please discuss the results on ACTG175 datasets and the different pattern observed in Fig3.c.

- Provide more empirical evidence on real-world data.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14