Externally Valid Policy Evaluation from Randomized Trials Using Additional Observational Data
Authors: Sofia Ek
Dave Zachariah
Venue: NeurIPS 2024 poster (2024)
Keywords: Policy Evaluation
Randomized Trials
External Validity
Primary area: machine_learning_for_healthcare
TL;DR: We propose a nonparametric method that uses trial data, adjusted with additional covariates from the target population, to provide certifiably and externally valid policy evaluations.

Abstract: Randomized trials are widely considered as the gold standard for evaluating the effects of decision policies. Trial data is, however, drawn from a population which may differ from the intended target population and this raises a problem of external validity (aka. generalizability). In this paper we seek to use trial data to draw valid inferences about the outcome of a policy on the target population. Additional covariate data from the target population is used to model the sampling of individuals in the trial study. We develop a method that yields certifiably valid trial-based policy evaluations under any specified range of model miscalibrations. The method is nonparametric and the validity is assured even with finite samples. The certified policy evaluations are illustrated using both simulated and real data.

Externally Valid Policy Evaluation from Randomized Trials Using Additional Observational Data
Sofia Ek
Dave Zachariah
Uppsala University sofia. ek0it. uu
Uppsala University dave zachariahcii uu
Abstract
Randomized trials are widely considered as the gold standard for evaluating the effects of decision policies. Trial data is, however, drawn from a population which may differ from the intended target population and this raises problem of external validity (aka. generalizability). In this paper we seek to use trial data draw valid inferences about the outcome of policy the target population. Additional covariate data from the target population is used to model the sampling of individuals in the trial study. We develop method that yields certifiably valid trial-based policy evaluations under any specified range of model miscalibrations. The method is nonparametric and the validity is assured even with finite samples. The certified policy evaluations are illustrated using both simulated and real data.
Introduction
Randomized controlled trials (RCT) are often considered to be the 'gold standard' when evaluating the effects of different decisions more generally, decision policies. RCT studies circumvent the need to identify and model potential confounding variables that arise in observational studies. They enable the evaluation and learning of decision policies for in, e.g., clinical decision support, precision medicine and recommendation systems (Qian and Murphy 2011 Zhao et al. 2012] Kosorok and
Laber 2019).
However, RCTS sample individuals that may differ systematically from a target population of interest. For instance, clinical trials usually involve only individuals who do not have any relevant comorbidities and those who volunteer for trials may very well exhibit different characteristics than the target population. Invalid inferences about decision policy can be potentially harmful in safety- cafetclcritical applications, where the cautionary principle of "above all, do no harm" applies (Smith) (005). This is especially challenging since the distributions of population characteristics unknown. How generalize results from the trial sample to the intended population??
The focus of this paper is the problem of establishing externally valid inferences about outcomes in target population, when using experimental results from trial population (Camphell and Stanley 1963 Manski 2007 Westreich] 2019). We consider evaluating decision policy, denoted π, that maps covariates X an individual onto a recommended action A. The outcome of this decision has an associated loss L (aka. disutility negative reward). Thus L may directly represent the outcome of interest. We assume the availability of samples (X, A, L) from the trial population and additional covariate data X from the target population (Lesko al. 2016 Li al. 2022 Colnet et al.) 2024).
The covariate data is used to model the sampling of individuals in the trial study. We propose method for evaluating policy that
is nonparametric and makes no assumptions about the distributional forms of the data,
takes into account possible covariate shifts from trial to target distribution, even when using miscalibrated sampling models with unmeasured selection factors,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

100%
50%
1.5
Mancrerraee gup
0.0
0006
-0.1
40%
0.2
2006
-0.3
0.2
0.6
Ea
Target
(a)
(b)
Figure 1: Inferring the out-of-sample losses policy æ. (a) The loss is bounded by an upper limit la with probability of least a. The RCT-based limit curve uses only trial data, whereas the other limit curves also utilize a sampling model trained using additional covariate data X from the target population. Each limit curve in blue is certified to provide valid inferences for models miscalibrated up to a degree I' defined in 3 (b) Gap between the actual probability of exceeding the limit, La and the nominal probability of miscoverage a. A negative gap means the inference La is invalid, while positive gap implies it is valid but conservative. Details of the experiment are
presented in Section 5.1
and certifies valid finite-sample inferences of the out-of-sample loss, up to any specified degree of model miscalibration.
Many policy evaluation methods are focused on estimating the expected loss Ex [4] of T. However, since substantial portion of losses may exceed the mean, this focus miss important tail events (Wang et al., 2018 Huang et al. 2021). By contrast, evaluating a policy in terms of its out-of-sample loss provides more complete characterization of its performance and is consonant with the cautionary principle. Figure iillustratt the evaluation of π using limit curves which upper bound the out-of-sample loss L with given probability 1 A limit curve based RCT-data alone only ensured to be valid for a trial population. Using additional covariate data, however, we can certify the validity of the inferences for the target population up to any specified degree of miscalibration of the sampling model.
The rest of the paper outlined follows. We first state the problem of interest in Section 2 and relate it to the existing literature in Section We then propose a policy evaluation method in Section and demonstrate its properties using both synthetic and real data in Section We conclude the paper with discussion about the properties the method in Section and its broader impact in Section 7
Problem Formulation
Any policy æ, whether deterministic randomized, can be described by distribution p., (A) Each covariate X, unmeasured selection factor and action A has an associated loss (o0, Lmaxx). We consider here discrete action space, i.e. A (1..., K}. The decision process has causal structure that can be formalized by directed acyclic graph, visually summarized in Figure 2 (Peters) al. 2017 The sampling indicator S indicates whether individuals are drawn from largel population, = 0, or a trial population, 5 1. The causal structure allows us to decompose the two distributions. Specifically, the target distribution factorizes as
(X, U, A, L/S = 0) = p(X, = 0) (AIX) U(L)X,U,A), where only the policy P- (AIX) is known. Similarly, the trial distribution factorizes as p(X,UJS 1) p(A[X) p(L U,
(1)
p(X, U, A, L/S
(2)
where only randomization mechanism p(A[X) known. In general the characteristics of target and trial populations may differ, that is, p(X,U'S = 0) / x(X, S 1). The unmeasured U may include self-selection factors that are challenging to record. Note, however, that only factors that also affect the loss L are relevant here.

---

5-0
S=1
X
X
A
(a)
(b)
Figure 2: Causal structure of process (a) under policy т as well as (b) the trial study. Sampling
indicator S distinguishes between the two. For the important case of RCT, assignment of A is not influenced by any covariates so that the path X + eliminated.
From the trial distribution (2) we sample m individuals D ((X;, A,, I.)) independently. In addition, we also obtain n independent samples of covariate-only data (X1. X X.)) from the target population (T). Our aim to infer the out-of-sample loss Ln+1 for individual
under any policy æ. Specifically. we seek loss limit la as function of - such that L I la holds with probability a as illustrated by the 'limit curves' in Figure Ta
The sampling pattern of individuals is described by P(S) X, U). This distribution is unknown, but fitted using held-out data S,)
assume that model X() is available. This model employing either the conventional logistic model or any state-of-the-ar art machine learning models (as exemplified below). It can also be obtained from previous studies. There is, however, guarantee that X(S]X) is calibrated and it may indeed diverge from the unknown sampling pattern. Nevertheless, we want inferences about the out-of- out-of-sample sample loss be valid also for miscalibrated models. We therefore express the degree of miscalibration in terms of the selection odds:
I I
p = 0|x,
p = 0|X)
<
I
(a.s.)
1($=1X,U)
P=S= 1|X)
odds
nominal
That is, the nominal selection odds can diverge by a factor I, where implies a perfectly calibrated model. This model includes all sources of errors (selection bias, model misspecification, estimation error).
limit e provides an externally valid inference of La+1, up to any specified degree of miscalibration T, if it satisfies
Pa (TT+1 S (T(D) 2 aa Va.
The problem we consider is to construct this externally valid limit ef This limit allows us to infer the full loss distribution of a future individual with L,+1 under policy n. rather than merely the expected loss [Z]. Specifically, the tail losses are important in healthcare and other safety critical applications where erroneous inferences could be harmful, and a cautious approach when implementing new policies is needed.
The limit curve e for policy is valid up any declared degree of odds miscalibration T, which establishes the credibility of the policy evaluation, cf. Manski (2003). While is unknowable, especially under unmeasured U, measured data (Rosenbaum and Rubin 1983 Tan (0066. Following the method in Huang (2024). we treat observed selection factors in X as unmeasured U in (3) to benchmark appropriate values for Т, as detailed in subsection 4.1
ideas from sensitivity analysis to guide its selection using
By increasing the range of r, certify the validity of the inference under inereasingly credible assumptions on X(S[X) (Manski 2003). As the model credibility increases, however, the informa- tiveness of the inferences decreases. Since the upper bound on the losses, Lmaxs is trivial and uninformative limit, we may define the informativeness of as
Informativeness inffa (D) < Lmaxx..
(5)
3

---

10055
100%
90%6
80%
- 1.5
60%
40%
00%
20%
20%
2.0
0%
1.5
2
10
odia( x.. x. Xo)/olds(X...)
(a)
(b)
Figure 3: (a) Omitting measured selection factors to benchmark credible values for r in (3. (b) Inferred blood mercury levels [ug/L] in target population under 'high' and 'low' seafood consump- tion 1 and No, respectively). Limit curves for degrees of odds miscalibration I E [1,2].
That is, the right limit of limit curve, which decreases with the degree of miscalibration. Figure La shows curves that are valid for miscalibration in the range I' E [1, 2]. where the informativeness is 95% and 90%. respectively. The latter figure means that can infer non-trivial bound on the loss for 90% of the target population.
Remark: This paper addresses the evaluation of given policy, whether proposed by clinical expert or learned from historical data. By setting aside samples from an RCT study, a policy т can learned, and its out-of-sample performance evaluated using the proposed methodology.
Background Literature
The problem considered herein is related to the problem of causal inference when combining data from randomized controlled trials and observational studies. Examples of the setting where only
covariate data from the observational study is available can be found in Lesko et al. (2016) and Li al. (022). For additional examples, refer to the survey by [Colnet al. (202)). Within the broader area of generalizability and transportability, the problem represents the case were the sampling probability depends solely on the covariates X, and is independent of the action and the loss (Pearl and Bareinboim, 2014 Lesko et al. [017; Degtiar and Rose 2023). The problem is also related to broader literature of statistical learning under covariate shifts, see for instance Shimodaira (2000); Sugiyama et al. 2007); Cittoonero-Candel et al. (2008); Reddi et al. (2015): Chen et al. (2016).
The most common object inference policy evaluation the expected loss E,, [L] and a popular method for estimating it is inverse probability of sampling weighting (IPSW), which models covariate shifts from trial target populations. The estimator using RCT-data defined as
! Σ D 1|x,)
P(S x)X;)
(r Xx)) p(A.)
ViPSw
I.j.
(6)
This methodology has been applied in various studies, for example Cole and Stuart 2010);
: Stuart
et al.] 2) Westreich et al. (2017); Buchanan et al. (2018). It is widely recognized that misspecified logistic models can introduce bias when estimating the weights (Colnet et al. (224) and more recent works have suggested using flexible models, such as generalized boosted methods (Kern et al. 2016), instead. The counterpart IPSW, used in off-policy evaluation with observational data, is inverse propensity weighting (IPW). The problem with misspecified logistic models also applies here when estimating the classification probabilities, aka. propensity scores (McCaffrey et al. [2004] [ee et al.) 2010). In this case, generalized boosted methods and covariate promising alternatives (Setodji et al., 2017 Tu 2019). The literature on IPSW and IPW mainly focused on average treatment effect estimation, that is E,, [Z] Er,, [Z] where ক and denote the 'treat all' and 'treat none' policies, respectively. By contrast, we want certify the distributional properties of Ln+1 for any n, even under miscalibration of X)S)
Dovariate-bllancin balancing methods have shown to be

---

Conformal prediction is distribution- free methodology focused on creating covariate-specific prediction regions that are valid for finite-samples (Vovk et al. 2005 Shafer and Vovk 2008). The methodology was extended by Tibshirani et al. (2019) to also work for known covariate shifts. Jin jet al. (2023) combined the marginal sensitivity methodology developed in Tan (2006) with the conformal prediction for covariate shifts to perform sensitivity analysis of treatment effects in the case of unobserved confounding. Our methodology draws upon techniques in conformal prediction. but instead of providing covariate-specifie prediction intervals under policy æ,, we are concerned with evaluating any т over a target population.
When full identification of the causal effect is not possible due to unmeasured confounders, partial identification sensitivity analysis can be used evaluate the robustness of the estimates. [Rosenbaum] and Rubin (183)) introduced sensitivity parameter to bound the odds ratio of the probability of treatment. More recent work has extended this approach to account for treatment effect heterogeneity and non-binary treatments, as seen in Tan (2006); et (201 1); /Zhao et (2019): Dorn and Guo (2023) among others. However, it is often challenging to interpret the absolute value of the sensitivity parameter in practical scenarios. To address this, recent research has proposed benchmarking, or calibrating, results by estimating the effects of unmeasured confounders, see for example Hsu and Small (2 Franks et al.) (2019); Veitch and Zaveri (2020); De Bartolomeis et al. (2024). Note that all these papers work with sensitivity analysis for observational studies. Huang (2024) instead combines sensitivity analysis for generalization with benchmarking to determine reasonable values of T.
A biased sample selection, similar to that described by was considered in the context of average treatment effect estimation by Nie et al. (202T). In contrast to that work, our primary focus is on ensuring the validity of inferences regarding out-of-sample losses, even when dealing with finite training data. We achieve this using a sample-splittin technique.
4 Method
Here we construct a limit (D) on the out-of-sample losses under policy that satisfies (4) for any given specified degree of miscalibration l'.
4.1
Benchmarking degree of miscalibration
The limit curve (D) holds up to the specified degree of odds miscalibration r. Although I is inherently unknown, sensitivity analysis and methods for assessing the calibration of models can guide its estimation using available data.
We start with a benchmarking method that specifically accounts for the potential impact of unmea- sured selection factors, U. Building on the approach in Huang (2024), we treat observed selection factors in X as proxies for unmeasured U in equation (3), providing benchmark for selecting suitable values. Specifically, let the the omitted selection factor Xg be U and X_ denotes the
remaining covariates. Then the ratio in 3 is estimated by dividing odds(X Xx) = p(5 by
odds(X-.) -(-00 - .) Figure 3a summarizes the distribution of odds(XX Xx)/ /odd//ddd/(e// (x
using real data set, where the omitted Xx is either age, income or education. We see that the corresponding ratios all fall within r = 2. We can therefore conclude that if any unmeasured selection factor U is weaker than the omitted factors, The corresponding loss curves are shown in Figure (3b] which illustrates the blood mercury levels in a target population under policies of 'high' respective "low" seafood consumption that can credibly be extrapolated from trial data. More details are available in Section 5.2 Note that this benchmarking method serves as a guide for assessing the influence of unmeasured selection factors U (Cinelli and Hazlet 2019).
credible to set in the range of 1.5 2.
Without unmeasured selection factors, methods for assessing the calibration of models X) discussed in, e.g., Murphy and Winkler (1977); Naeini et (2015); Widmann el al. (2019) can guide the specified lower bound of the range of miscalibration. The nominal selection odds in (3), i.c..
odds(X) = x() can be quantized into several bins and for each bin the unknown selection
odds, i.e., odds(X) x) can be estimated by counting the samples from both the target and
p(S=0)X)
trial distributions present. In the case of well-calibrated model, estimated unknown odds should
5

---

match the quantized nominal odds for each bin. To assess calibration, this process can be iterated across multiple ranges within the dataset and visualized through reliability diagram, as exemplified in Figure Using (3), we have that
(X)s(X) I odds(X) <
r (X)ssX).
Take the expectation with respect to X, conditional on the nominal odds in a specified interval (or bin) I, so that:
1
r E(odds(X)) odds(X) E 7] I ((odds(X)) odds(X) I] I r EEooddd (X)
odds(X) E I].
The expected odds is then estimated for each bin I by counting samples from the target and trial distributions.
4.2 Inferring out-of-sample limit
We will now construct the limit (D). For this we need to describe the distribution shift from trial to target distribution for all samples, including the n + sample. We begin by considering the true distribution shift expressed using the ratio
(X U, A,
U
Inserting the factorizations T and 2 into this ratio shows that specifying the distribution shift requires the unknown (conditional) covariate distribution p(X, UIS). We can, however, bound using the model of the sampling pattern, X)) as follows:
c.W < Px (X,U, U, A, LJS 0)
p(X, U, A, L = W.
(8)
where
1 P(S = 0|X) (AI
W = PS 0|X) (+[X
W
p = 1[x p(A[X)
p(A
S(== and c p[==
is constant. To see this, we note that the ratio in can be expressed as
p(s -S==0XXU)) (A(x)
p(S = 1X, p(A[X)
using Bayes' rule. The bound (8) follows by applying (3). We proceed to show that the factors (
are sufficient to construct an exterally valid limit e" for odds divergences up to degree I, similar to Ek et al. (2023).
To ensure finite-sample guarantees, the trial data is randomly divided into two sets, D = D' UD", with respective samples sizes of m' and m'. The set D' is used to construct
w r [To
(m' 1)(1 3) < m',
(D'
(10)
x0,
otherwise,
where l is the upper limit in 99) evaluated over D' and ordered W [2] S .... Wl
[m/] We
show that 10) upper bounds the ratio 77) for future sample with probability - 3 for any choice
of B E (0, 1). The set D" is used to construct a stand-in for the unknown cumulative distribution function of the out-of- loss:
EIED" W 1 (I, 5)
F(E; D", w) =
(11)
EIEDO [W s 2) + W. 1(L, > 2)] +
where 0 is a free variable representing the unknown out-of -ample weight W n+1 for a future sample. Based (10 and define ET as the quantile function
C = inf { {e:
1-a
F(t; (D)) 2 1-3
(12)
This enables us to construct valid limit on the future loss T for any miscoverage probability (0, 1).

---

Algorithm 1 A set of limit curves for policy x
input Model D(SIX) policy Pa (A X)), trial policy pX) set of miscalibration de- grees {1. T max and trial data D.
Thw Til
output {(T, 0. 20
2.0
1: Randomly split D into D' and DD"
1.5
2: for TorTe ( mox do
1.0
3: for a E {0,
do
4: for 3 ( a)
0.5
5: Compute wy as 10
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Neminnl adds
6: Compute 0,0 as in 12 end for
7:
Figure 4: Reliability diagram of the observed odds against the average predicted nominal odds obtained from models (S)
8: Set E to the smallest e
9: Save (T, ax, (.).
10: end for
11:
end for
Theorem 4.1. For any odds miscalibration up to degree r,
(D)
min et¹ 8:<$<< a,B:
(13)
is an externally valid limit on the out-of out-of-sample loss Lnt1 of policy That is, satisfy
13 is certified to
The method seeks the level / for the bound (10) that yields the tightest limit. The proof is presented in the supplementary material and builds several techniques developed in Vovk et al. (2005);
Tibshirani et al. (2019); Jin et al. (2023); et al. (2023).
Algorithm summarizes the implementation of set of limit curves given a model of the sampling pattern (SS and
of miscalibration degrees [1, max]. Note that (TO) and (IT) are step
functions in 3 respective f. Therefore (12) and (13) can be solved by computing the functions at
grid of points. Calculating (10) and
requires the sorting of weights, but the sorting operation is
one-time requirement.
Increasing the degree of miscalibration I' results in a decrease in W and an increase in W in (9). As weights associated with lower and higher losses decrease and increase, respectively, in the resulting limit E (D) becomes more conservative.
Remark 1: If the trial population a small subgroup of the target population (for example in the case P(S=0[X)
of weak overlap), the nominal odds p(S= I[X] will tend to be high. As this yields significant weight (DD'. the informativeness (5) of a (D) diminishes. Nevertheless, the guarantee in { holds.
Remark 2: The split of 'D can be performed in a sample-efficient manner in the case of RCT, where actions are randomized so that x(A(X) = p(A): For the ith sample, (X,, Ai, Li), draw A; ~ Pm A X,). Include sample in D" if A; = A,, otherwise include it in D'. This sample splitting method ensures that inferences on the loss are based on actions that match those of the policy.
5 Numerical Experiments
will use both synthetic and real-world data to illustrate the main concepts of policy evaluation with limit curves (ox, (.). As performance benchmark, we estimate the quantile which yields the tightest limit using the inverse probability of sampling weighting (Colnet et al. 2024
(D) = inf {E: Fipsw (; 2 - -a},
(14)
where
m x)= (A(A)))))) 1(L, S $)
FiPSw (e;
p(=1Xx)) p(A.)

---

Table 1: Means and variances covariate distribution p(X,U|)) (15)
Population
P0.5
P1,8
P0,3
0
o
U.
A ( = 0)
0.5
0.5
0.5
1.0
1.0
1.0
B (S
0.0
0.5
0.0
1.25
1.5
1.25
Trial (S = 1)
0.0
0.0
0.0
1.0
1.0
1.0
Logste
Xiliooi
102
Logatk
NGBoos
-2
-2
-2
10
10
10nt
-2.5
0.0
2.5
2.5
0.0
2.5
-2
0.0
2.5
-2.
0.0
2.5
-25
0.0
2.5
-2.5
0.5
2.5
(a) Selection odds for target population A.
(b) Selection odds for for target population B.
Figure 5: Odds p(S OjX)/p(S l[X) compared with nominal odds obtained from logistic and XGBoost models X))) The dots are a random subsample of the trial samples.
This is similar to the approach in Huang et al. (2021) but adapted to problems involving data from trial and observational studies.
We examine the impact of increasing the credibility of model assumptions, i.e., by increasing the miscalibration degree I, on the informativeness (5) of the limit curve. In addition, for the simulated data, we also assess the miscoverage gap of the curves
Miscoverage gap = a Pa(In+1 > (D()))==))
where a negative gap indicates an invalid limit.
5.1 Illustrations using synthetic data
We consider target and trial populations of individuals with two-dimensional covariates, distributed follows:
Xo.s N
|o.s |1,s
Too
0 2
X
oi.s
UIS (pp.s.oo.s).
(15)
X
where the parameters are given in Table to be unknown.
[ The distributions for populations A, B and Trial are taken
The actions are binary A E {0, 1 and corresponds to 'do not treat' versus 'treat'. We evaluate the 'treat all' policy, i.e. Pr. (A 1|X) The trial an RCT with equal probability of assignment, i.., p(4) = 0.5. The unknown conditional loss distribution is given by
LI,,,X,U) ~ AAAA x + A U + + (1-11 1),
The sampling probability p(S]X, U) is treated unknown. The complete of hyperparameters used is provided in the supplementary material.
For the observational data (x,)" we drew n 2000 samples. For the trial data we drew 1000 samples: m 500 samples and ((X,, A., L.)) was used to compute the limit curves. The remaining 500 samples were used to train ((S)X).
Figure compaares nominal selection odds obtained from the fitted models with the unknown odds -[S 1|X) for target population A. We consider two different fitted models p(S) X):
logistic model, which conventionally used the causal inference literature /Westreich et al. 2017).
and the more flexible tree- bee-based ensemble model trained by XGBoost (Chen and Guestrin 2016).
In this case, the logistic model happens to be well-specified so that learned odds approximate the true ones well. The more flexible XGBoost model also provide visually similar odds, albeit less accurate. By contrast, Figure(5b)eppp repeats the same exercise for target population B. Here the logistic model is misspecified and severely miscalibrated while the XGBoost model continues to provide

---

100%
Cuiltted conuriate XL
80%
N2
Miscoverage gap
607
0.0
-0.1
2056
Benchmark
1.0
-02
2.5
3.5
0.0
02
0.4
0.8
1.0
odias X Xo)oddds(X..)
Target
(a)
(b)
Figure 6: Evaluating a 'treat all' policy Tı for target population B. (a) Benchmarking the degree of miscalibration I' using omitted covariates. (b) Miscoverage gaps when degree of miscalibration Te E [1.
visually similar odds. A well-performing and flexible model is required for meaningful benchmark of the upper value of T. Therefore, we will use the XGBoost model.
In Figure we use the reliability diagram technique to assess the performance of the XGBoost model odds(X] X) of the nominal odds for target population B. The model is close to the diagonal suggesting that it sufficiently flexible to accurately model the odds. This is in line with the result in Figure (bb
We then use observed covariates to calibrate appropriate upper bounds for Г. Figure (6a shows the evaluation with respect to population B. Assuming that the unmeasured U have no greater selection effect than Xo. degree of miscalibration I 2 is credible choice as it covers more than 95% of the ratios. Since the data is simulated, we evaluate the miscoverage gap of the limit curves of the 'treat all' policy in Figure 6b for the benchmark and the limit curves for the proposed method. The gap is estimated using 1000 independent runs and for each run drawing 1000 independent new samples (Xn+1. Un+1. An+1 Ln+1). We see that the benchmark and the limit curve for I 1 yields a negative miscoverage gap. As the degree miscalibration increases to 2, the limit curves exhibit positive miscoverage gaps.
The evaluation of Tı with respect to population shown Figure Results for the logistic model, another policy To, and for additional populations are available in the supplementary material.
5.2 Evaluating seafood consumption policies
To illustrate the application of policy evaluation with real data, we study the impact of seafood consumption on blood mercury levels with data from the 2013-2014 National Health and Nutrition Examination Survey (NHANES). Following Zhao et 2019) each individual's 's data includes eight covariates, encompassing gender, age, income, the presence or absence of income information, race, education, smoking history, and the number of cigarettes smoked the last month. All covariates, except for smoking history U, are treated as measured and denoted X. We excluded one individual due to missing education data and seven individuals with incomplete smoking data. We impute median income for 175 individuals with income information. The continuous covariates age, income and number of cigarettes smoked in the last month - are standardized. After the preprocessing, data set comprises 1107 individuals. The data is then split into observational data Do and trial data D, based on the covariates gender, age, income and smoking history (more details are available in the supplementary material) resulting in 646 samples in the observational data and 461 samples in the trial data. The action A describes individual fish or shellfish consumption, categorizing an individual as having either low (51 serving in the past month) or high (> 12 servings in the past month) consumption. The loss L represents the total concentration blood mercury (measured in μg/L).
To generate counterfactual actions and losses, we consider a balanced RCT so that pl ()) 0.5 and learn model of p( L,A,X from the data using gradient boosting (Freund and Schapire 1997

---

Friedman et al. 2000). Thus during training, the trial data consists of samples (X,, A,, Li) whereas the observational data only contains X.
In Figure Sa we use observed covariates to benchmark appropriate upper bounds for r. We exclude each of the seven covariates one by one, highlighting the three most dominant ones in the figure. If the unmeasured covariates have weaker influence than these, 1" value in the range of 1.5 to 2 is appropriate. In Figure 3b we compare the limit curve for policy wo corresponding low (A = 0) seafood consumption with the limit curve for a policy T corresponding to high (A = 1) seafood model po( ()).. For reference, a mercury level of μg/L is
consumption. We use an XGBoost- guidance limit for women of child-bearnng age. We see that under a low consumption policy a lower mercury level can be certified for miscalibrated odds I [1, 2]. In this case, we can infer a 90% probability that the out-of-sample mercury level falls below the reference value of pg/L.
6 Discussion
We have proposed a method for establishing externally valid policy evaluation based on experimental results from trial studies. The method is nonparametric, making no assumptions on the distributional forms of the data. Using additional covariate data from a target population, it takes into account pos- sible covariate shifts between target and trial populations, and certifies valid finite-sample inferences of the out-of-sample loss Ln+1, up to any specified degree of model miscalibration.
Conventional policy evaluation methods focus on E [4] and can easily introduce a bias without the user's awareness, particularly when the model of the sampling pattemn ((S]X) is misspecified. Lacking any control for miscalibration undermines the possibility to establish external validity. In safety-critical applications, making invalid inferences about a decision policy can be potentially harmful. Hence, adhering to the cautionary principle of "above all, do no harm" is important. The proposed method is designed with this principle mind, and the limit curve represents the worst-case scenario for the selected degree of miscalibration r.
We also exemplify how the reliability diagram technique and the benchmark approach of omitting observed selection factors facilitate a more systematic guidance on the specification of the odds miscalibration degree I' in any given problem.
7 Broader Impact and Limitations
The method we propose is designed for safety-critical applications, such as clinical decision support, with the cautionary principle in mind. We believe that it offers valuable tool for policy evaluation in such scenarios. Our approach focuses on limit curves, coupled with statistical guarantee, for more detailed understanding of the out-of-sample loss. This facilitates a fairer evaluation by bringing attention to sensitive covariates in the tails. However, it remains important to be aware of biases, and might be necessary to address them separately to prevent their replication. note that the method requires independent samples, condition that may not be met during major virus outbreaks or similar situations.
also important to
Acknowledgments and Disclosure of Funding
This work partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and the Swedish Research Council under contract 2021-05022.
References
Ashley I. Buchanan, Michael G Hudgens, Stephen R Cole, Katie R Mollan, Paul E Sax, Eric S Daar, Adaora A Adimora, Joseph Eron, and Michael Mugavero. Generalizing evidence from randomized trials using inverse probability of sampling weights. Journal of the Royal Statistical 2018
Society Series A: Statistics in Society,
Donald T Camphell and Julian Stanley, Experimental and Quasi-experimental Designs for Research. R. McNally College Publishing Company, 1963.
10

---

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart. Robust covariate shift regression. In Artificial Intelligence and Staristics, pages 1270-1279. PMLR, 2016.
Carlos Cinelli and Chad Hazlett. Making Sense of Sensitivity: Extending Omitted Variable Bias.
Journal of the Royal Statistical Society Series B: Statistical Methodooogy, 82(1):39-67, 12 2019. ISSN 1369-7412. doi: 10.1111/rssb. 12348. URL 0ttp::::///0i..rr/10111111111111 11110....11..sr.. 12348
Stephen R Cole and Elizabeth A Stuart. Generalizing evidence from randomized clinical trials to target populations: the actg 320 trial. American journal of epidemiology, 172(1):107-115, 2010.
Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaäl Varoquaux, Jean- Philippe Vert, Julie Josse, and Shu Yang. Causal inference methods for combining randomized trials and observational studies: a review. Statistical science, 191) 2024.
Piersilvio De Bartolomeis, Javier Abad Martinez, Konstantin Donhauser, and Fanny Yang. Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. In International Conference on Artificial Intelligence and Statistics, pages 1045-1053. PMLR, 2024.
Irina Degtiar and Sherri Rose. A review of generalizability and transportability. Annual Review of Statistics and Its Application, 10:501-524, 2023.
Jacob Dorn and Kevin Guo. Sharp sensitivity analysis for inverse propensity weighting via quantile balancing. Journal of the American Statistical Association, (66444):2645-2657 2023.
Sofia Ek, Dave Zachariah, Fredrik D Johansson, and Peter Stoica. Off-policy evaluation with out-of-sample guarantees. Transactions on Machine Learning Research, 2023.
Centers for Disease Control and Prevention (CDC). National Center for Health Statistics (NCHS). National Health and Nutrition Examination Survey Data. Hyattsville, MD: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, 2013. URL https
//wwwn www...cdc..c. http://hhh/maaaaeesc/ooiiiiiiiia
AlexanderM Franks, Alexander D) Amour, and Avi Feller. Flexible sensitivity analysis for observa- tional studies without observable implications. Journal of the American Statistical Association, 2019.
Yoav Freund and Robert E Schapire. A decision decision-theorctic theoretic generalization on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-13) 1997.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2): 337-407, 2000.
Jesse Y Hsu and Dylan S Small. Calibrating sensitivity analyses to observed covariates in observa- tional studies. Biometrics, 69(4):803-88 811, 2013.
Audrey Huang, Liu Leqi. Zachary Lipton, and Kamyar Azizzadeneshelii Off-policy risk assessment in contextual bandits. Advances in Neural Information Processing Systems, 34:23714-23726, 2021.
Melody Y Huang. Sensitivity analysis for the generalization of experimental results. Journal of the Royal Statistical Society Series A: Statistics in Society, page qnae012, 03 2024.
Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6). 2023.
Holger I. Kern, Elizabeth A Stuart, Jennifer Hill, and Donald P Green. Assessing methods for gener- alizing experimental impact estimates to target populations. Journal of research on educational effectiveness, 91):103-127, 2016.
11

---

Michael R Kosorok and Eric B Laber. Precision medicine. Annual review of statistics and its application, 6:263-286, 2019.
Brian K Lee, Justin Lessler, and Elizabeth A Stuart. Improving propensity score weighting using machine learning. Statistics in medicine, 29(3):337-346, 2010.
Jing Lei, Max G G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-fre free predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111, 2018.
Catherine R Lesko, Stephen R Cole, H Irene Hall, Daniel Westreich, William C Miller, Joseph Eron, Jianmin Li, Michael Mugavero, and CNICS Investigators. The effect of antiretroviral therapy on all-cause mortality, generalized to persons diagnosed with hiv in the usa, 2009-11. International journal of epidemiology, 45(1): 150-150, 2016.
Catherine R Lesko, Ashley L. Buchanan, Daniel Westreich, Jessie K Edwards, Michael Hudgens, and Stephen R Cole. Generalizing study results: potential outcomes perspective. Epidemiology (Cambridge, Mass.), 28(4):553, 2017.
Fan Li, Ashley L Buchanan, and Stephen Cole. Generalizing trial evidence to target populations in non-nested designs: Applications to aids clinical trials. Journal of the Royal Statistical Society Series C: Applied Stutistics, 71(3):669-697, 2022.
Charles Manski. Identification problems in the social sciences and everyday life. Southern Economic Journal, 70(1): 2003.
Charles F Manski. Identification for prediction and decision. Harvard University Press, 2007.
Daniel F McCaffrey, Greg Ridgeway, and Andrew R Morral. Propensity score estimation with boosted regression for evaluating causal effects observational studies. Psychological methods, (4):403, 2004.
Allan H Murphy and Robert L Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society Series C: Applied Statistics, 26(1):41-47, 1977.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. Proceedings of the AAAI conference on artificial intelligence, 29 (1), 2015.
Xinkun Nie, Guido Imbens, and Stefan Wager. Covariate balancing sensitivity analysis for extrapolat- ing randomized trials across locations. arXiv preprint arXiv:2112.04723,, 2021.
Judea Pearl and Elias Bareinboim. External Validity: From Do-Calculus to Transportability Across Populations. Statistical Science, 29(4):579 - 595, 2014. doi: 10. 10.1214/14-ST5486
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Min Qian and Susan Murphy. Performance guarantees for individualized treatment rules. Annals of statistics, (9(2):1180) 2011.
Joaquin Quinonero-Candee Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. Pro- ceedings of the AAAI Conference Artificial Intelligence, 29(1), 2015.
Paul R Rosenbaum and Donald B Rubin. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society: Series (Methodological), 45(2):212-118, 1983.
Claude M Setodji, Daniel F McCaffrey, Lane F Burgette, Daniel Almirall, and Beth Ann Griffin. The right tool for the job: Choosing between covariate balaneing and generalized boosted model propensity scores. Epidemiology (Cambridge, Mass.), 28(6):802, 2017.
12

---

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Changyu Shen, Xiaochun Li, Lingling Li, and Martin C Were. Sensitivity analysis for causal inference using inverse probability weighting. Biometrical journal, 53(5):822-837, 2011.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):27-2244, 2000.
Cedric M Smith. Origin and uses of primum non nocere-above all, do no harm! The Journal of Clinical Pharmacology, 35(4)3371-377, 2005.
Elizabeth Stuart, Stephen R Cole, Catherine Bradshaw, and Philip Leaf. The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society Series Statistics Society, 174(2):369-388) 2011.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe.
Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural information processing systems, 20, 2007.
Zhiqiang Tan, distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619-16)) 2006.
Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Chunhao Tu. Comparison of various machine learning algorithms for estimating generalized propen- sity score. Journal of Statistical Computation and Simulation, 89(4):708-719, 2019.
Victor Veitch and Anisha Zaveri. Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. Advances in neural information processing systems, 13:10999-11009 2020.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science & Business Media, 2005.
Lan Wang, Yu Zhou, Rui Song, and Ben Sherwood. Quantile-optima treatment regimes. Journal of the American Statistical Association, 113(523) 1254, 2018.
Daniel Westreich. Epidemiology by Design: A Causal Approach to the Health Sciences. Oxford University Press, Incorporated, 2019. ISBN 9780190665760.
Daniel Westreich, Jessie K Edwards, Catherine R Lesko, Elizabeth Stuart, and Stephen R Cole. Transportability of trial results using inverse odds of sampling weights. American journal of epidemiology, 186(8):00001100 2017.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in neural information processing systems, 32, 2019.
Qingyuan Zhao, Dylan S Small, and Bhaswar B Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Staristical Society: Series B (Statistical Methodology), 81(4):735-70 2019.
Yingqi Zhao, Donglin Zeng, A John Rush, and Michael R Kosorok. Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 1066419) 2012.
13

---

Supplementary Material
In this supplementary material, we provide the proof outlined in Section A and additional details on the numerical experiments discussed in Section B
A Proof
and Ps (X, U, A, S 0). The proof technique presented here builds upon several results established in
In this context, I represents the probability over samples drawn from both
Vovk et al. 2005): Tibshirani al. (2019); Jin et al. (2023); Ek al. (2023). and for completeness
we present the proof in full.
Following
Ek et al. (2023) we introduce B such that 0 < < a. Use
12
to construct the limit
(D) inf { e: :: F(e (D')
1 1-
(16)
1-
where TH(D" is defined in (10). We want to lower bound the probability of Ln+1 S (D). Note that PfIn+1 & (D)) = (( S (d) W Wht1 S (D)) P( n+1 S (5(D)) +P[Ln+1 S (D) W .+ TH(D') P( W a+1 > (D)))
from the law of total probability. The second term is lower bounded by zero, and we have
P(In+1 < (D)) 2 P(L.+I S (D) w n+ I W/(D")) n+1 S wg((D)} (17)
Let us focus on the second factor in w (D') lower bounded by
From the construction in (TO) the probability of W n+1 S
P W P(W
S (F(D')) 2 1
(18)
Vovk et al. 2005) Lei al. 2018 thm. 2.1).
We now proceed to bound the first factor 17 i.e., P(In+1 S Sd((D) Wh S wD((D)). Define
the following limit
(D"", Wht in int F(E,""" W W
1 1- a
(19)
1- 3
where W n+1 2 Wn+ is given in 8. Comparing this limit with the one defined in (16), we see that
P(Ln+1 S ed (D) W n+1 (D') P(Ln+1 S ED" Wh+1) W n+1 I T
=P(L,41 I e." w 11+1)).
whenever W n+1 & wh (D'). The second line results from applying sample splitting, which gurantess that Ln+1 < (D" W n+1 and WT n+1 S. WJ(D') are independent.
To lower bound PfLn+1 S (D", W } we will make use of the following inequality,
=E E [(C, W } EiED" W s ( + W 1(4 e W a+1
E W 1L $ +
1
3
(20)
that holds by construction.
First, define S4 as unordered set of the following elements
((Xw+1, Un +1, Am +1, Lmn .... (Xms Um. Am Lm), (Xn+1s Unt1, Anti, Lnt1)).
14

---

From Tibshirani et al. (2019 thm. 2) we have that the out-of-samppl loss L,+1 has the (conditional) cdf
P(Ln+1 S8| [+] E p 50)= Eies, w; 5 Eies, wy
(21)
iES,
where wy quantifies the distribution shift for sample using the (unobservable) ratio Next, build on the proof method used in Jin et al. (2023) thm. 2.2). Use the limit (D" w Wn+1) from (19
in (21) and apply the law of total expectation perform marginalization
S+
P(Ln+1 S (D" W m+1)] = [ S " m+11) S+]]
Eiss, W,1(L, - (D", W. m+1)) Eks, W
B
(22)
We proceed to establish lower bound for this probability. Combining 20 and 22 have that
P(L++1 S (D"" W w++1]
1-a a 1- B 2 ΣiEs, WLI(L, & (5)
EiEDo W. S (.))
E
Exes, W
EiED"
WF S ( + W 1(4 > ( + n+
(+)
=E
Eies, Wi
EiEDi W 1(4 S + +1 (g) +
where
(+) Σ WilLL 2t
Σ 11 > ( +
LES_
LED"
I (())
Σ Wi > Eg)
IED!!
1E8
WL [ [5))
Σ
WL > (.)) Wat1
LiED"
WIl(1+ S {
Σ
Wil(L, > ( W
IED"
LEED"
We use the bounds provided
to derive the inequality. Hence, we arrive at a valid limit
P(Ln+1 S (D" W Wh+1) w++1))+ 2 1-a
(23)
1- 3
Finally combine (18 and 23
to get
P(L++15 (D)) 2 P[ < (D" ,W" P(W S (D()'))
> 3 (1 3)
a
(24)
a.
We choose in 13 get the tightest limit. As Ton+1 is drawn from P- (X,U, A, T.S. = 0), we express 24 shown in (4) for the sake of notational clarity.
15

---

Table Hyperparameters used for XGBoost in Section 5.1
Parameter
Value
n_estimators
100
max_depta
2
learning_rate
0.05
objective
binary:logistic
whid_wiid_
subsample
0.6
bysseeel__bytree
0.8
colsamplee_yleevll
0.4
scale_pos_weight
Ts== o/ n1
Table Hyperparameters used for XGBoost in Section 5.2
Parameter
Value
n_estimators
50 1
max_depth
learning_rate
0.2
objective
binary:logistie
win_chiild_eiight
subsample
0.4
bytree
0.7
bessampe__blleee
0.8
faalle_os_wwiihee
12=00/ 71==1
B Numerical Experiments
Additional information of the numerical experiments outlined in Section can be found in this supplementary material and the code used for the experiments can be accessed here http://ww.....t..a////oo//e//aa.
https:///itthb..commm wtt.hh/w....../.....eeeeeeeeeeae
All experiments were conducted using Version 2.0 License). A comprehensive list of hyperparameters is available in Table for the synthetic case and Table 3 for the NHANES case. The hyperparameters were selected through random search involving 200 runs, employing 5-fold cross-validation with the Fl score the optimization metric. All experiments were performed on a laptop with the following specifications: Intel Core i7-8650 CPU @ 1.9GHz, 16 GB DDR4 RAM, and Windows 10 Pro 64-bit operating system. The experiments utilized only the CPU, The total time required to run the experiments was approximately half an hour.
the Python implementation XGBoost (Apache-
B.1 Synthetic data
We extend the experiments in Section 5.1 to include two extra target populations C, respective D. The full list of parameters used in (15) are given in Table
To illustrate the generality of the proposed methodology, we consider two different fitted models X(S] logistic model, which conventionally used in the causal inference literature, and the more flexible tree-based ensemble model trained by XGBoost. Figure compares nominal sampling odds obtained from the fitted models with the unknown odds, p(S S x)X)/p(SS 1|X), for the target populations C, respective D. This corresponds to a case without unmeasured selection factors U. In all these cases, the logistic model is misspecified and miscalibrated while XGBoost model provides odds that resemble the true ones. Note that the proposed algorithm can handle any model (3) is satisfied. However, a well-performing model generally required to achieve small I and for meaningful benchmark of the upper value of l'.
16

---

Table 4: Means and variances covariate distribution D(X,U|S) (15)
Population
P0.5
P1,8
P0,S
o
U
A ($ = 0)
0.5
0.5
0.5
1.0
1.0
1.0
B (S
0.0
0.5
0.0
1.25
1.5
1.25
C ($ = 0)
0.0
0.0
0.0
1.5
1.5
1.5
D (8
0.25
0.25
0.25
1.0
0.25
0.5
Trial (S 1)
0.0
0.0
0.0
1.0
1.0
1.0
True
Logistic
XGBoost
10a
2
D
100
10-2
-2
2
10
10-6
350
(a) Sampling odds for for target population C.
True
Logistic
XGBoosi
10a
2
0
109
10-2
-2
-2
10-4
10-6
-2
2
-2
0
2
Zn
Zig
Zo
(b) Sampling odds for target population
Figure 7: Odds p(S = OpS) = 1|X) compared with nominal odds obtained from logistic and XGBoost models X)) The dots a random subsample of the trial samples.
In Figure use the reliability diagram technique described in subsection 4.1 to assess and compare the performance of logistic and XGBoost models of the nominal odds, odds (X), for the synthetic data. For all numerical experiments, we use bins. For target population A in Figure Sa] both models are close the diagonal and it is reasonable to believe that they are flexible enough to model the odds. For the other three populations, B, C and D, in Figure Sb 8c respective 8t] is evident that the XGBoost model shows a closer alignment with the diagonal compared the logistic model. This is in line with the results in Figure and where the XGBoost model is visually closer to the true model.
We now use the observed covariates to benchmark appropriate upper bounds for I' for all target populations as described in subsection 4.1 Figure Pa shows the evaluation with respect to population Assuming that the unmeasured selection factors U of similar strength the weakest covariate, X1, a I' value of 2.5 could be credible choice, as it covers 90% of the odds ratios. Figure 9b shows the same evaluation with respect to population A I value of 1.2 could be credible choice for the logistic regression model and a I' value of 2 could be appropriate for the XGBoost model. However, from Figure 5b we know that the logistic model misspecified in this scenario. For population in Figure 9c and 2.5 seem to be suitable I values for the logistic regression model respective the XGBoost model. Similarly, for population
Figure 9d 1.5 and 2.2 could be suitable values for the two models. The logistic model is again misspecified in populations C and D.
17

---

1.0
Legjatic XcBBook
Logasic
3.5
NGBooot
3.0
P 1.5
3.0
P 1.5
Tiee odla
2.5
2.5
2.0
1.5
1.0
1.0
0.5
Nominal odda
Nominal odds
(a) Reliability diagram for target population
(b) Reliability diagram for target population B.
4.0
Logistie
Logistic
3.5
XCBooet
3.5
XGBoooet
3.0
P- 1.5
3.0
F- 1.5
Thue cdik
2.5
2.0
1.5
1.5
1.0
0.0
0.5
1.5
2.0
3.0
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.5
Nominal orlds
Nominal odlds
Reliability diagram target population
(d) Reliability diagram for target population D.
Figure 8: Reliability diagram of the observed odds against the average predicted nominal odds obtained from logistic and XGBoost-trained models XXS)X).
In Figure 10 we use the limit curves for the benchmark in 14 and the proposed method to evaluate the out-of-sample loss of the 'treat all' policy 3i, i.e.
Ps. (A = X) 11
Figure 10a shows the evaluation with respect to population A. When r = all the limit curves are similar. The curves for the proposed method also illustrate increasing the credibility of the models results in less informative inferences. However, their informativeness stays above 90% for odds miscalibration degrees E [1, 2). We also evaluate the miscoverage gap of the curves and observe that the benchmark and the limit curves for I have negative miscoverage gap. As the degree of miscalibration increases to 2, the limit curves exhibit positive miscoverage gaps, where XGBoost results in slightly less conservative inferences than the logistic model does. We continue with population B and in Figure [Ob] respective [0c The limit curves derived for the baselines closely align with the curves modelled using logistic regression when I 1, but is consistently lower than the curves modelled using XGBoost. For the certified curves the informativeness stays above 90% for odds miscalibration degrees r E [1, For the miscoverage gap, the baselines and the limit curves for = are invalid. As the degree of miscalibration increases to 2, all limit curves indicate positive miscoverage gaps. In Figure [Od] we evaluate the same for population D. The limit curves modelled using the baseline and logistic regression when I = infer consistently higher losses than the curve modelled using XGBoost. For the curve using the logistic model the informativeness stays above 90% for odds miscalibration degrees I [1.2]. The same figure for the XGBoost model is approximately 95%. For the miscoverage gap. the baseline and the limit curves for I are invalid. Again, as the degree of miscalibration increases to 2, all limit curves indicate positive miscoverage gaps.
We now turn to comparing the 'treat all' policy with a 'treat none" policy, i.e.,
Pao (A 0 X) 11
18

---

100%
100%
Omiitoed cottriate X
Omiiteed covoraate
Ng lngistir
lagietie
80%
X6 Xghoost
80%
Xgboost
X1 logietic
logistic
X1 Xghoost
Xgboost
60%
60%
40%
40%
20%
20%
0.5
22.0
3.5
1.5
3.0
oddaa( X Ay Xo3/oddat X s)
oddsl X Jo Xi3/odds X
Benchmarking for population A.
(b) Benchmarking for population
100%
100%
Omnitzzd conoriate Xg
Omiised cotoriate X2
logietic
80%
Xg sgboort
Ng riboowt
logistic
logietic
60%
Xg xgboust
60%
Xgboost
F 2
F" 2
40%
40%
20%6
2056
099
0%6
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
33.5
4.0
oddds(X X Xx3/oddis(XX
odddo[ X Xg3/oddat X
x)
(c) Benchmarking population C.
(d) Benchmarking for population
Figure 9: Benchmarking the degree of miscalibration uusing omitted covariates.
for population A. Their expected losses are estimated using (6) as Vipsw 1.64 and Vipsw = 1.54, respectively. This evaluation suggests that Hi preferable to Tu- However, the limit curves presented in Figure [ provide a more detailed picture in terms of out-of-sampll losses: the tail losses certified for the 'treat none' policy To are lower than those certified for лj. This illustrates the cautionary principle built into the policy evaluations. Similar results are observed for target population D. While target populations and C, both ViPSw and Vipsw exhibit comparable sizes, the proposed limit curves still offer a more detailed understanding of out-of-sample losses.
For completeness, the actual degree miscalibration for all target populations visualized in Figure 12 for the case without unmeasured selection factors U.
B.2 Seafood consumption policies
The National Health and Nutrition Examination Survey data (NHANES) is produced by federal agencies and is the public domain, allowing it to be reproduced without permission. our evaluation the data is split into observational data Dn and trial data DD based the covariates age, income, gender and smoking history (age and income are standardized), e.g.
P(S 1|X,0) U) = 0.25- 0.25-[/((x))) ([(gge)) ) (Xincome)) +0.05 + [1 (Xmale = 1)+ 1 smoking = +03,
where f()) is the sigmoid function.
19

---

10058
80%
0.2
Misowenage gap
6098
0.1
0.0
Bendhmakk
Tngietic, = 1.0
Logistic, I 2.0
-0.1
2055
NGBoat,
XGBoost, r 2.0
-0.2
09%
0.0
0.4
0.6
0.8
Targot
(a) Limit curve and miscoverage gap for targel population A.
10096
80%
Msoowwaae gap
80%
0.1
0.0
Benchmark
lagisti, T- 1.0
Legietic, = 20
-0.1
2055
XGBoont F 1.0
WGRoost 2.0
0.2
096
0.0
0.2
0.4
01.
0.8
1.0
Targot
(b) Limit curve and miscoverage gap for target population B.
10058
80%
Mesewergge gap
00%
Benehmack
Lngietic, T 1.0
Leghtic, - 20
-0.1
2056
XGRoodt == 1.0
XGDoost, 2.0
-0.2
0%
0.0
0.2
0.1
0.6
0.8
1.0
Targot
(c) Limit curve and miscoverage gap for target population C.
1005%
80%
Mercwerage gap
40%
Benchmarks
Tegistic, T
1.0
Logistic, 20
-0.1
205
XGBond, - 1.
XGBoost 2.0
-12
0%
-2
10
0.0
0.4
0.6
0.8
Target
(d) Limit curve and miscoverage gap for target population D.
Figure 10: Evaluating 'treat all' policy æi for different target populations with degrees of miscalibra- tion I' E [1.
20

---

100%
100%
80%
80%
60%
2.0
60%
20/%
086
0%%
-2
2
6
-2
4
(a) Limit curves for target population
(b) Limit curves for target population B.
100%
100%
80%
80%
1.0
2.0
2.0
40%
40%
20%
20%
09
09%
-22
(c) Limit curves for target population
(d) Limit curves for target population
Figure 11: Limit curves for and Ti for different target populations certified for I E [1. 2].
Logistie
XCiBoos
Logistie
XoBoo
107
167
107
30
30
10
20
Fg
(a) Degree of miscalibration for target population A.
(b) Degree of miscalibration for target population B.
Logistie
XCDDooot
Logistio
XCBoost
104
10
100
109
10-1
10-1
-2
-2
(c) Degree of miscalibration for target population
(d) Degree miscalibration target population D.
Figure 12: The actual degree of miscalibration for different target populations. The dots random subsample of the trial samples, where the green ones correspond to a degree of miscalibration within T and the black ones to degree of miscalibration not bounded by r 2.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main assumptions are presented in Section the method is detailed in and the experiments are described in Section
Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The assumptions are included in Section 2 and some limitations are raised in Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: Main assumptions are available in Section 2 the method is described in Section and the proof in the supplementary material
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experiments are described in Section with more details in supplementary material B
The code is also provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is available the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ crapp..cc/pblli/guuu.ee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiments are described in Section 5 with more details the supple- mentay material
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The miscoverage gap is reported, validating the statistical guarantee of the method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Included in the supplementary material
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics http::/ttttttsis.ccceeo////////e http////////tttttcccctceeeeelele
didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and this research conforms to its guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Included in Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Included in the supplementary material
B
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "Rp4FNevJ1w", "reviewer": "Reviewer_Cyms", "summary": "This paper studies the challenge of generalizing randomized controlled trial (RCT) results to a target population, addressing the potential issue of distributional shift from RCT participants to the intended population. Instead of estimating the expected loss on the target population, the paper proposes a nonparametric method that leverages covariates information from the target population and certifies valid finite-sample inferences of out-of-sample tail loss. The approach is validated using both synthetic and real data, ensuring its applicability to the target population.", "questions": "Please refer to the weakness part for questions.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written. For me who work on causal inference but not very familiar with the methodology on handling distributional shift, I can easily follow the paper and get the gist of the idea. \n2. I think the method and the technical results are very clean and sound. \n3. I believe the inference on out-of-sample tail risks is of vital importance, which is often omitted in policy evaluation.", "weaknesses": "1. If I understand correctly, the policy Pi is given as an exogenous parameter. However, it is a common practice that we learn a policy using the RCT data and aim to know its performance on the target population. This definitely introduces the dependence issue. I suspect some type of cross-fitting may help but I doubt the empirical performance, or any chance to analyze the bias in this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the question and are pleased that he/she acknowledges the importance of the problem we are tackling.\n\n>  If I understand correctly, the policy Pi is given as an exogenous parameter. However, it is a common practice that we learn a policy using the RCT data and aim to know its performance on the target population.\n\nYes, this paper focuses on the *evaluation* of any given policy $\\pi$, which could either be proposed by any clinical expert or *learned* using past data. It is therefore possible to set aside, say, $N$ past samples from an RCT study to learn a policy $\\pi_N$ and then use the proposed methodology to evaluate its out-of-sample performance. We will add a remark in the manuscript to inform the reader about this possibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oCM8j9DOuN", "reviewer": "Reviewer_S2kx", "summary": "This paper aims to use trial data to make valid inferences about policy outcomes for a target population. By incorporating additional covariate data from the target population, the sampling of individuals in the trial study is modeled. The authors develop a nonparametric method that provides certifiably valid trial-based policy evaluations, regardless of model miscalibrations, and ensures validity even with finite samples. The effectiveness of the certified policy evaluations is demonstrated using both simulated and real data.", "questions": "NA", "limitations": "Yes. I have very limited knowledge for this area, my reviews for this paper should not be count.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Paper has theoretical analysis.", "weaknesses": "1. paper is not well organized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s honesty and trust that the chair will discount this review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohsqAGUD3g", "reviewer": "Reviewer_grGf", "summary": "This paper proposes a method for constructing limit curves (upper bounds on the CDF) of an outcome, under a given policy, in a target population, using data from an experimental study.  The general goal is to certify that bad outcomes are unlikely in a target population, given experimental data and some knowledge of the strength of potential biases arising from selection on unobservable characteristics, model misspecification, and so on.", "questions": "I would appreciate any clarifications or additional comments from the authors on the two main weaknesses I raised above:\n1. How to incorporate model misspecification or estimation error into the choice of Gamma?\n2. Are there any parts of the technical contribution the authors would highlight as particularly interesting from their perspective?  Honest question, as I have not read some of the related papers in depth.", "limitations": "The authors adequately discuss limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Full disclosure: I reviewed a previous version of this paper, and previously recommended rejection. Hence, my review is influenced both by the current version, and by my knowledge of what has changed since the previous version.\n\nWithout further ado, the strengths of this paper, as I see them:\n1. This paper considers an important and significant problem at the intersection of several interesting lines of work (e.g., conformal prediction & risk control, generalization of policy evaluation from experimental to observational settings, etc).\n2. With some minor nits (see below), the paper clearly presents their contributions, the motivation for their approach, and the assumptions required.  Moreover, the current version does a much improved job putting some of their work in the context of other papers that consider similar bounds on generalizing from experimental to observational data.\n3. The approach is practical, given the incorporation of some informal methods for benchmarking plausible values of the $\\Gamma$ parameter.  This is new since the previous version, and greatly improves the practical applicability of the method in my view.", "weaknesses": "The main weakness, in my view, is the claimed contribution that Gamma can incorporate finite sample and model misspecification error (see lines 77-78, \"includes all sources of errors...selection bias, model misspecification, estimation error)\"), since it is not at all clear how those considerations enter into the selection of Gamma?  The informal benchmarking approach involves building intuition for plausible impacts of selection bias due to unobserved factors, but it didn't seem to speak to model misspecification or estimation error.  This is not a major issue necessarily, but it might be worth softening some of the claims that the proposed approach handles all these other types of error, without some corresponding method for benchmarking these.  For instance, for smaller sample sizes, I could imagine these factors being much more influential in the true value of $\\Gamma$.\n\nThe second weakness, which I'm less inclined to weigh heavily, is that the novelty of technical contribution is slightly unclear: Mechanically speaking, there is little difference between sensitivity analysis considering unobserved confounding that impacts selection & outcome, versus confounding that impacts treatment & outcome, and there is similarly little difference between policy evaluation and evaluating average treatment effects.  It would be helpful if the authors could highlight the technical contributions they think are most notable compared to Jin et al. 2023, Ek et al. 2023, Huang 2024, etc, beyond the simple fact that the setting differs (e.g., considering treatment-outcome vs selection-outcome confounding, or considering ATE vs policy evaluation).\n\nI also have some additional minor feedback that might be worth incorporating into a future version:\n1. Regarding the benchmarking, it would be worth giving the caveat that this is an informal approach to benchmarking that can yield unintuitive results, see [0].  Part of the contribution of Huang 2024 (cited here) was to give a more principled approach under a different sensitivity model.  However, I don't view this as a major weakness, since this is generally an unresolved challenge as I understand it for Rosenbaum-like sensitivity bounds as used here (Huang 2024 & Cinelli and Hazlett 2020 use an R2-based sensitivity model where a more principled approach is possible in the first place).\n2. There's a lot of work on combining observational & experimental data, and it might be worth highlighting differences to the settings considered. For instance, [1] deals with a similar setting, but one where outcome information is available from the observational data. There are other papers that deal with estimating causal effects in target populations from experimental data where observational data includes individuals not represented in the trial, e.g., [2], [3].\n3. It's not entirely clear from the introduction what the \"loss\" $L$ entails, as the name suggests something like a prediction error.  I think it would be useful to highlight upfront that $L$ can represent e.g., an outcome of interest (more typically rendered as $Y$), as done in the application.  It is more intuitive to me to interpret the setup in light of the application, e.g., the goal is to bound the probability of extreme outcomes occurring.\n4. As an improvement, it seems like it should be fairly straightforward to derive lower bounds as well, no?  E.g., replacing $L$ with $-L$ and applying the same machinery.  That might be useful in applications where you want to ensure that an outcome stays within a certain range.\n\n[0] Making Sense of Sensitivity: Extending Omitted Variable Bias.  Carlos Cinelli, Chad Hazlett. https://doi.org/10.1111/rssb.12348 (JRSS B, 2020)\n\n[1] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. Piersilvio De Bartolomeis, Javier Abad, Konstantin Donhauser, Fanny Yang. https://arxiv.org/abs/2312.03871 (AISTATS 2024)\n\n[2] Removing Hidden Confounding by Experimental Grounding.  Nathan Kallus, Aahlad Manas Puli, Uri Shalit. https://arxiv.org/abs/1810.11646 (NeurIPS 2018)\n\n[3] Falsification before Extrapolation in Causal Effect Estimation. Zeshan Hussain, Michael Oberst, Ming-Chieh Shih, David Sontag. https://arxiv.org/abs/2209.13708 (NeurIPS 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the reviewer’s past comments, which improved the revision substantially. \n\n> The informal benchmarking approach involves building intuition for plausible impacts of selection bias due to unobserved factors, but it didn't seem to speak to model misspecification or estimation error. This is not a major issue necessarily, but it might be worth softening some of the claims that the proposed approach handles all these other types of error, without some corresponding method for benchmarking these.  [Q1]  How to incorporate model misspecification or estimation error into the choice of Gamma?\n\nThis is a valid point: the benchmarking method addresses only the plausible impact of unobserved selection factors $U$. We will add a remark that points the reader to Appendix B.3, where we suggest using reliability diagrams to quantify bounds on model misspecification or estimation errors.\n\nThe possible joint impact of $U$ *and* model misspecification or estimation error is, of course, beyond the scope of our methodology.\n\n> The second weakness, which I'm less inclined to weigh heavily, is that the novelty of technical contribution is slightly unclear.  [Q2] Are there any parts of the technical contribution the authors would highlight as particularly interesting from their perspective?\n\nYes, while the `mechanical’ aspects of the employed proof technique does build on Jin et al. 2023 and cited works developed in other problems areas, the contribution to the problem of establishing externally valid policy evaluation is novel. We have tried to break down our proof steps in Appendix A in a transparent manner and highlight those steps that invoke past results (e.g. lines 767, 776, 778). To make it clearer we will also refer to the relevant theorems in the cited work.\n\nRegarding the minor feedback:\n\n> [1] Regarding the benchmarking, it would be worth giving the caveat that this is an informal approach to benchmarking that can yield unintuitive results, see [0]. Part of the contribution of Huang 2024 (cited here) was to give a more principled approach under a different sensitivity model. However, I don't view this as a major weakness, since this is generally an unresolved challenge as I understand it for Rosenbaum-like sensitivity bounds as used here.\n\nWe agree. While we have tried to be transparent about it, adding a reference could make it even more formal.\n\n> [2] There's a lot of work on combining observational & experimental data, and it might be worth highlighting differences to the settings considered. For instance, [1] deals with a similar setting, but one where outcome information is available from the observational data. There are other papers that deal with estimating causal effects in target populations from experimental data where observational data includes individuals not represented in the trial, e.g., [2], [3].\n\nWe have descriptions of the setting both in the introduction (lines 25-31) and the background (lines 109-117), but the difference to other settings may not be clear enough. We will review the references and try to clarify this.   \n\n> [3] It's not entirely clear from the introduction what the \"loss\" $L$ entails, as the name suggests something like a prediction error. I think it would be useful to highlight upfront that $L$ can represent e.g., an outcome of interest (more typically rendered as $Y$), as done in the application. It is more intuitive to me to interpret the setup in light of the application, e.g., the goal is to bound the probability of extreme outcomes occurring.\n\nWe refer to it as “loss” as the opposite of “reward”, either way it is an *ordered* decision outcome that we want to bound. However, this is a good suggestion that will help the reader and we will try to clarify it already in the introduction.\n\n> [4] As an improvement, it seems like it should be fairly straightforward to derive lower bounds as well, no? E.g., replacing $L$ with $-L$ and applying the same machinery. That might be useful in applications where you want to ensure that an outcome stays within a certain range.\n\nReplacing it with $-L$ will result in a lower bound. Although we have not considered this type of application, it is a valuable suggestion and we will add a remark about it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C69XnQPlAC", "reviewer": "Reviewer_xLhe", "summary": "This paper introduces a method for inferring policy decisions based on randomized controlled trial data when applied to a target population with new covariate data. The method is nonparametric and makes no assumptions about the distributional forms of the data and certifies valid finite-sample inferences of the out-of-sample loss.", "questions": "1. Could you clarify the rationale behind inferring L_{n+1} for a single additional data point rather than considering the distribution of outcomes for multiple new observations (1...n)?\n2. Is there a way in your experiments to evaluate the estimated p(S|X) and hence odds ratios (Figure 4)? Is proposed alg 1 sensitive to mis-estimation of p(S|X), which is very likely due to unobserved confounding U?", "limitations": "The authors have marked some limitations, such as the method requires independent samples and may not be suitable in scenarios like major virus outbreaks.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem of policy evaluation with covariate shifts studied in this paper is important.", "weaknesses": "1. The motivation for problem setup is not clear. Why inferring L_{n+1} for the additional (n+1)th data point rather than the new 1...n? This would be reasonable in clinical trial, n new patents coming in with unknown outcomes.\n2. Discussion of theoretical guarantees is not thorough, especially when the method relies heavily on heuristics, e.g. strong assumptions about the correctness of the estimates of unknown distribution of p(S|X,U) are made, and division of D into D' and D'' seems arbitrary.\n3. The experimental setup looks overly simplistic, a basic two-d covariate space and a simple quadratic model for the loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and provide our response below. We believe they may clarify some potential misunderstandings.\n\n> [W1] The motivation for problem setup is not clear. Why inferring $L_{n+1}$ for the additional $(n+1)$th data point rather than the new $1...n$?\n\nBounding $L_{n+1}$ quantifies the out-of-sample performance of $\\pi$ with respect to a *future individual* drawn from the target population (after having sampled the covariates of $n$ persons), see lines 81-83.\n\nIn our assessment, it could be possible to extend the methodology to consider, say, $N$ future individuals, i.e., $n+1, …, n+N$ by leveraging permutations of independent samples. However, it appears to lead to a combinatorial explosion which quickly becomes intractable. We have therefore opted for clear intuitive results that are computationally feasible.\n\n> [W1, cont’d] This would be reasonable in clinical trial, $n$ new patents coming in with unknown outcomes.\n\nIt seems this could be a misunderstanding of the problem setting we are considering: The aim of the paper is not to infer outcomes in a *clinical trial*, but rather outcomes of a policy applied to a *target* population for which we only have covariate data.\n\n> [W2]  the method relies heavily on heuristics, e.g. strong assumptions about the correctness of the estimates of unknown distribution of p(S|X,U) are made, and division of D into D' and D'' seems arbitrary.\n\nA *central* point of the methodology we develop is precisely to *avoid* strong assumptions about $p(S|X,U)$ (see for instance lines 34-35, 69-80, 131-132, 290-291). We do so using a model $\\hat{p}(S|X)$ and the sensitivity specification $\\Gamma$ in eq. (3).\n\nSample splitting $\\mathcal{D}$ into $\\mathcal{D}'$ and $\\mathcal{D}''$, as described in lines 168-169, is a standard procedure in statistical inference to ensure valid inferences. (In supervised machine learning, this occurs e.g. in train-test splits to evaluate out-of-sample risk of a learned prediction rule.)\n\n> [W3] The experimental setup looks overly simplistic, a basic two-d covariate space and a simple quadratic model for the loss.\n\nThe first experimental setup presented in Section 5.1 is indeed intended to be *illustrations* as the subheading indicates. This is so that the reader can obtain some intuition behind the method. For this reason we could either use one or two-dimensional covariates to visualize the setting. We chose two-dimensional covariates as in Figure 4.\n\nNote that the method uses a *nonparametric* assumption for the loss distribution; and only uses a parametric model for the sampling pattern, i.e., $\\hat{p}(S|X)$, along with $\\Gamma$. The simplicity of the conditional loss distribution is therefore not relevant when illustrating the method.\n\nThe second experimental setup presented in Section 5.2 (and Figure 3) evaluates seafood consumption policies based on real data and involves 8 covariates. This is a considerably more complex problem for policy evaluation.\n\n> [Q1] Could you clarify the rationale behind inferring L_{n+1} for a single additional data point rather than considering the distribution of outcomes for multiple new observations (1...n)?\n\nPlease see our reply to W1 above.\n\n> [Q2] Is there a way in your experiments to evaluate the estimated p(S|X) and hence odds ratios (Figure 4)? Is proposed alg 1 sensitive to mis-estimation of p(S|X), which is very likely due to unobserved confounding U?\n\nWe are unsure about the first part of the question. Figure 4 belongs to the illustrative experiments in Sec. 5.1 in which there is a synthetic ground truth $p(S|X)$. In real experiments, our suggestion is to use ideas from sensitivity analysis (see lines 86-99 respective Figure 3a and 5a) to evaluate a credible range for $\\Gamma$ that limits the odds ratios in eq. 3. \n\nThe second part of the question seems to restate the misreading above: A *central* point of Alg. 1 is that it is *robust* against mis-estimation of $p(S|X)$, not least when there is unobserved confounding $U$! Hence the (benchmarked) miscalibration degrees $\\Gamma$ required as input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper considers a very important problem: leveraging randomized trial data and partial knowledge of treatment selection to ensure that bad outcomes are unlikely in a target population. As noted by grGf and Cyms, the paper is well-written, the methodology is well-motivated and practical, and the authors have clearly contextualized their contribution within the existing literature. That said, I must still strongly encourage the authors to revise their paper along the lines suggested by grGf. Specifically, please clarify the limitations of handling both misspecification and confounding from a benchmarking point of view and please clarify which theorems are being used from the cited references. 

Note: I have ignored the review and score of S2kx, who noted that their review should not count due to their limited knowledge of this area. I believe that xLhe's initial lower assessment was based, at least in part, on a misunderstanding of the problem setting. The authors have clarified this point satisfactorily. So my recommendation is based primarily on the reviews of Cyms and grGf.
Decision: Accept (poster)