Off-Policy Evaluation with Out-of-Sample Guarantees
Authors: Sofia Ek
Dave Zachariah
Fredrik D. Johansson
Peter Stoica
Venue: Accepted by TMLR ()

Abstract: We consider the problem of evaluating the performance of a decision policy using past observational data. The outcome of a policy is measured in terms of a loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under a different and possibly unknown policy. Using a sample-splitting method, we show that it is possible to draw such inferences with finite-sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method takes into account model misspecifications of the past policy - including unmeasured confounding. The evaluation method can be used to certify the performance of a policy using observational data under a specified range of credible model assumptions.

Published in Transactions on Machine Learning Research (06/2023)
Off-Policy Evaluation with Out-of-Sample Guarantees
Sofia Ek
sofia. aeffa.ceOlll.u... uu..e
Department of Information Technology
Uppsala University
Dave Zachariah
dave.zacharich su.se
Department Information Technology
Uppsala University
Fredrik D. Johansson
Department of Computer Science Engineering
Chalmers University of Technology
Petre Stoica
ps uu.se
Department of Information Technology
Uppsala University
Reviewed on OpenReview: https: // http:///epervvee....m net/ forum? id= XnYtCPgC9p
Abstract
We consider the problem of evaluating the performance of decision policy using past observational data. The outcome of a policy measured in terms of loss (aka. disutility or negative reward) and the main problem is making valid inferences about its out-of-sample loss when the past data was observed under different and possibly unknown policy, Using sample-splitting method, we show that it is possible to draw such inferences with finite- sample coverage guarantees about the entire loss distribution, rather than just its mean. Importantly, the method lakes into account model misspecifications of the past policy including unmeasured confounding. The evaluation method can be used to certify the performance of policy using obscrvational data under specified range of credible model assumptions.
1 Introduction
In this work, we are interested in evaluating the performance of decision policy, denoted n, which chooses action from discrcte action set. Each action A is taken in context with observable covariates X and incurs real-valued loss L (aka, disutility negative reward). Such policies considered, for example. in contextual bandit. problems and precision medicine (Langford & Zhang 2007 Qian & Murphy 2011 Lattimore & Szepesvári, 20200: Tsiatis et al.) 2019). For instance, A may be one of several treatment options for patient with observable characteristics X, and L mcasures the severity of the outcome.
A target policy æ can be evaluated using experimental data obtained from trials. Such experiments are, however, often costly to perform and may lead rather small sample sizes in, e.g,, clinical settings. Moreover, in safety-critical applications, it is often unethical to test policies without severe restrictions on the trial population well the target policy. A more fundamental inferential problem, however, is the lack of 'external' validity, i.c., the limited ability to extrapolate from the trial population to the intended target population may lead to invalid inferences (Westreich, 2019 Manski] 2019). The main alternative is off- policy evaluation, ie., using observational data from a past decision process to infer the performance of the target policy. For this to be valid one needs to assume that the past process modeled without systematic errors, i.c, no unmeasured confounding and using well-specified models. The credibility of these assumptions, therefore, determines the 'internal' validity of inferences about from observational data Manski 2003).

---

Published in Transactions on Machine Learning Research (06/2023)
Inferences that lack validity particularly serious when evaluating in decision processes that costly safety-critical. In such cases, even inferences that are asymptotically valid with increasing sample size may not be adequate. Moreover, when the resulting distribution of losses is skewed or is widely dispersed, its tails are important to evaluate. Then inferring the expected loss E,[L], as commonly done, provides a very limited evaluation of (Wang et al. 2018). For instance, the average loss a population may be small but the tail losses can be unacceptably large. such applications, we are more concerned with providing valid certifications of the overall performance (see Figure Iap, rather than inferences of single distributional parameter.
In this paper, we propose method for evaluating specified target policy using observational data that
provides finite-sample coverage guarantees for the out-of-sample loss,
evaluates the entire loss distribution instead of, e.g., its expected value,
and takes model misspecification, including unmeasured confounding, into account.
100%
100%
80%
Actual coverage
60%
60%
40%
40%
20%
20%
X:
0%
0.0
0.5
1.0
0%
20%
40%
60%
80%
100%
8
Target coverage
(a) Evaluation of out-of-sammle loss L.
(b) Evaluation of coverage of limit curves.
Figure 1: Evaluating out-of-sample losses under target policy with binary decisions A E {0, 1}. Policies To and correspond to 'treat none' (A = 0) and 'treat all' (A = 1), respectively, while denotes a policy that adapts to context X. (For more details, see the example in Section 5.1 with p1-) (a) Each curve certifies that a new loss L falls below the limit la with a probability of at least - aa using n = 1000 data points. The certified performance of policy π dominates those of the alternative policies. (b) Evaluation of actual coverage, that is, the probability of S la, versus target coverage 1
2 Problem formulation
We consider target policy for deciding an action A different contexts, which are described by observed and unobserved covariates X and U, respectively. The policy correspouds can be either deterministic random. Our aim is to evaluate the losses L that result from applying any given Each instance of contextual covariates, action, and loss, i..., (X, U, A, L.), is drawn independently from a target distribution Ux At our disposal is an observational data set
distribution P= (*(AAX] (A[X), and
D (Xx,, A,, L.))
and our goal to use it to characterize the out- of- sample loss Ln+1. Note that it was collected under a possibly differrntt policy than æ. If we continue with the example of patients, then In+1 would quantify the severity of the outcome for new future patient that is unobserved at the time our evaluation. Specifically, for any miscoverage level a E (0, 1), we seek informative limit f, (D) on the loss such that.
Po Lnti S (D() 2 1 - a.
(2)
2

---

Published in Transactions on Machine Learning Research (06/2023)
In other words, (D) a function of a yields finite-sample performance certification of π as is illustrated in Figure Ta Unlike a single point estimate, the limit curve characterizes the distribution of losses under w. For instance, consider a patient population where (X,U) denote its covariates, and A is its received treatment with an outcome loss L. Then across all coverage levels, (2) ensures that the treatment of a future patient will incur a loss no greater than (D) under policy x with confidence
a. Figure Tb shows the probability that a limit la (D), proposed below, bounds the future loss In+1 across all values of I a. Note that limit curves can be constructed for several alternative target policies, which enables us to compare and focus on the parts of their loss distributions that are most relevant to the specific decision problem.
The causal structure of this decision process is illustrated in Figure 2a.
The target distribution admits a
causal factorization
= Px ((X.U.AA)-(x,,)).....<< p4) p(IJA,X,U)
(3)
where p(X,U) U) and p(L|A, X,U) are unknown. The central challenge in aff-policy evaluation of π is that is not sampled from (3) but from shifted training distribution which admits a causal factorization
[
= -(X,
((LJA,
where p(A{X, U) characterizes a past policy (aka. behavioral policy) that may differ from x. The causal structure corresponding to 4 is illustrated in Figure 2b If the past policy were known, it would be possible to adjust for the distribution shift from training to target distribution using the ratio
Pa Px,XXU,,,) (=(A U(X,U,ALL P(A[X,U)
(5)
This is feasible in certain problems with fully automated decision-making, such online recommendation systems, where the past policy is designed using observable covariates only, i.e., |X U) = p(A In more general problems, however, we have only a nominal model of the past policy x(A[X) (aka. propensity model), typically estimated from prior observable data. The nominal model may therefore diverge from p(A[X, U) due to various modelling errors that persist even in the large-sample scenario: model misspecification and unmeasured confounding via U (Peters et al., 2017 Westreich 2019). Here we follow the marginal sensitivity methodology of Tan (2006) and characterize the model divergence with respect the odds of taking action A. That is, the nominal odds diverge from the unknown odds by some bounded factor I 2 1 as follows:
1
P(AIX,U) S I- U(AIX.U)
((AA)) XAAXX
ST,
(6)
F
unknown odds
nominal odds
for all discrete actions A and almost surely all X,U. When the bound equals I' = 1, the nominal model is perfectly specified and there is no unmeasured confounding. In general, we consider all cases where the nominal odds diverge by at most factor r. Note that (6) can accommodate all possible sources of errors in the nominal model [(A X), including model misspecification as well as finite-sample crrors.
In summary, the problem we consider is to construct a limit t. for target policy и using a nominal model X(A[X) and r. The resulting limit should satisfy the finite-sample guarantee (2) for all miscoverage levels a, and thereby cortify the target policy performance for any specified bound r in 66. This enables robust evaluation of larget policies using observational data since it can be performed across a range of credible odds divergence bounds I as we will illustrate in the numerical experiments in Section 5
By increasing the divergence bound r, the credibility of our assumptions on P(A[X) increases, but the informativeness of inferences about L,+1 decrease, cf. Manski (2003). Therefore we advocate to evaluate a policy a using several r in the range [1, max]: where I Fimxx is the maximum value for which the limit curve remains informative in any given problem. The informativeness of valid limit curve can be defined follows:
Informativeness 1 1--a" a", where a' = inf (D) < Lmax].
(7)
where Imax is the maximum value of the support of L. That is, the highest coverage probability at which we can informatively certify the performance of n, i.c. the right limit of curve. Note that the informativeness is a problem-dependent quantity.
3

---

Published in Transactions on Machine Learning Research (06/2023)
x
X
A
L
A
L
(a) Decision process under target policy
(b) Decision process under past policy
Directed acyelic diagrams representing the causal structure of the decision process under (a) target policy, which yields (3), and (b) past policy, which yields ( In (b), both contextual covariates (X, U) may
Figure 2:
joint affect actions A and outcome loss L. Thus U gives rise to unmeasured confounding.
3 Background
We place the problem considered in this paper and our proposed method within the framework of off-policy evaluation.
Expected loss: In the off-policy evaluat ion literature, the target quantity is often the unknown expected loss E, [ of policy æ. A standard estimator of the mean, that dates back to Horvitz & Thompson (1522) is based on inverse propensity weighting:
VIeW (D) -- - (X,,,A)) Li.
(8)
where A)
P. x)
is model of (5) see for instance Rosenbaum & Rubin (1983) Beygelzimer et al.
x)
(2099; Qian & Murphy 2011 Zhang et al. (0112) Zhao et al. (2012); Kallus (2018). We note that the
estimator in 18] is unbiased when r= 1. An alternative standard estimator is based on regression modeling:
VRM(D) TI
-
1
Σ iE
(9)
where X), is a model of X].
The approaches in (8] and have complementary strengths and weaknesses resulting from the challenges of modelling the past policy and the conditional well-specified, the accuracy of the ostimators depends highly the overlap of covariates X across decisions A in the training data (Oberst et al. 2020: D'Amour et al. 2021). When the overlap is weak, the variance of (D) can become excessively large, even when it is unbiased. This can be mitigated by clipping the
of losses, respectively. Even when the models are
weights, which in turn canses bias Rubin 2001 Kang & Schafer 2007 Schafer & Kang 2008 Strehl et al. 2010).
When the models ((X, A) or X) exhibit systematic errors, however, the corresponding estimators in and are biasod and may invalidate the evaluation of w. The 'doubly robust' estimator
VDR = - (w(X. X(A, + Σ (r(x)) ((a.ax)))(aaxx.)))
acA
is one way of robustification in the case of either model @(X, A) or ((A,X) misspocified. Morcover, it reduces the estimator variance provided {(A, X) is sufficiently accurate [Bang & Robins 2005 Dudík et al. 2011 Rotnitzky et al. 2012).
Distribution of losses: When the loss distribution is highly skewed and the tails are long, the use of expected loss to evaluate policies be inadequate, especially in high-stakes problems. There are alternative
4

---

Published in Transactions on Machine Learning Research (06/2023)
parameters of the loss distribution, described by the cumulative distribution function F(C) = P. Px(La+1 S
(odf). that one can consider in such problems, e-g-, quantile or the conditional value at risk (Wang et al. 2018 Chandak et al. 2021 Huang et al. 2021).
Off-policy evaluation of x with respect to some alternative parameter can be achieved using edf-cstimators that are analogous to the mean estimators above, see Huang et al. (2021) In analogy to 8 tte inverse propensity weighted cdf-estimator
(e:
1 n t0 oXx, 1(4, s 1),
Σ
(10)
is point-wisc unbiased when r = 1. Similar to 99 the estimator
(e:
((( i=
requires a model Zf;a, a, x) of the conditional distribution P[L S A, To mitigate against model mis- specification that threatens the validity of the evaluation of म, one can use the 'doubly robust' estimator
(DD(C: Σ
Pn (a) El; X.)), Σ ( - + Σ
aEA
which protects in the case that either model (XX, A) X(A,)) is misspecified. While this estimator is consistent, it is not guaranteed to yield proper edf.
In this paper, we are interested in limiting the out-of-sample loss Ln- The a-quantile is the smallest la such that Prr PriIn I la) 2 a. It can be estimated as
la (D) = inf {e: F(e;D) 2 21 a
using one of the odf-cstimators above. We will use Firw as a benchmark below.
Distribution-free inference: Derivations of fimite-sample valid, nonparametric limits on random variables date back to the works of Wilks [941]; Wald 1943) Scheffe & Tukey 1945). More recently, the related methodology conformal prediction has focused on developing covariate-specific prediction regions (Vovk) et al. 2005 Shafer & Vovk 2008 Vovk 2012) See Lei & Wasserman| (2014); Lei et al. (2018): Romano al. (2019) for further developments. Tibshirani et al. (019) extended the methodology to make it valid also under known covariate shifts. This extended methodology was used to provide context-specific prediction intervals for any given policy x, which are statistically valid under the assumption that the past policy p(A[X,U) U) is known (Osama et al. 2020 Taufiq et al. 2022).
The marginal sensitivity methodology developed in Tan 2006) enables us to specify a more credible range of assumptions using 6 This type methodology was used for robust policy learning Kallus Zhou (2021), for robust policy learning sequential decisions in Namkoong et al. (2020) and for sensitivity analysis of treatment effects in Jiin et al. (2023) in the of unobserved confounding. The present paper considers the overall performance of similar to Huang et al. (2021). However, different from that paper, we focus ensuring inferences on the out-of-sample losses that are valid even for finite training data and systematic modelling errors including unobserved confounding using a sample-splitinn technique that leverages
results derived in Jin et al. (2023
4 Method
We show that one can limit the out-of-sample losses for n under any specified odds divergence bound P 2 for the nominal model in (6) which we assume holds. It follows that is bounded as:
W < (X,UU, SW p(X,U, U. A,
(11)
5

---

Published in Transactions on Machine Learning Research (06/2023)
where the bounds equal
W = P=)(A)) (1-441))((1++77((44) (7(AI)
and W == [1TT()))))""X
(12)
The model X)) typically estimated from prior observable data. The above bounds are functions of and drawn from the training distribution 4. In order to ensure finite-sample guarantees, we randomly split the training data ① into two separate sets,
D= L1,
with samples of sizes no and no. respectively. The set Di is used to form the function
E (I, S
F(E; (1;
(13)
Σ Di=no+1 WL, S Di=n++1 W,1(L, > E) w
as a proxy for the unknown edf of the out-of-sample loss Ln+1 and w > 0 represents the ratio in [11) We use the set Do to construct an upper bound on this unknown ratio. As the following result shows, this enables construct valid limit l, on the future loss Ln+1-
Theorem 4.1. Define the following quantile function of 13
la.3 =in inf e: u3
1 1
2 1 - 3
(14)
where
W ([ooo+))(1-0111
[(no + +1)(1 1)(1-8))
B)1 nn,
(Dn)
(15)
o0,
otherwise,
and W [k] denotes the kth order statistic of (W W For any miscoverage probability E (0,1), construct
(a(D)
min la,s, 800BCa
(16)
which a valid timit on the out-of-sample loss La+1 with probability of at least as specified in
The proof of Theorem 4.1 can be found in Appendix A.1
The variable W (Do) in 15 provides a statistically valid upper bound on the unknown ratio in 111 where B only specifies its confidence level. For any given value of 0 < B < a, la, in (14) bounds the loss Ln+11
with a coverage probability of at least a. Consequently, (2 chooses the tightest achievable limit.
Remark When r ==1, 1, then W W [1]. Therefore 13 can be thought of as an inverse propensity
weighted edf, where the weights are normalized to sum unily.
We now turn to the implementation of 16 We nole that that we in (15) is a piecewise constant function in 3. Therefore 14 and subsequently 116 can be solved by evaluating functions along a discrete set of points.
13 is piecewise constant function in & and
using a discrete grid of miscoverage levels a. Given a model p (A and range of odds divergence bounds T, Algorithm produces set of corresponding limit curves.
The computation of a limit curve is summarized in Algorithm 1
5 Numerical experiments
In the experiments below, we evaluate policies using the limit curves (a, ta). We quantify how increasing the credibility of our model assumption, i.e., by increasing 1, affects the informativeness of the limit curve using (7.. We also consider the coverage probability of the curves:
Miscoverage gap
P- (Tnt1 (D))
(17)

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm Limit curve of policy
Input: Policy Px (A[X), training data D, model ((A[X), bound r 2 1 and sample split size no.
1: Randomly split D into Do and D1.
2: for a E {O...., } do
3: for 3 a do
4: Compute W using 15 Compute la using 114 end for Set la to the smallest la above. Store (a, end for Output: {())aa)
When this gap is positive, the limit is conservative and when the gap is negative the limit is invalid, respectively, at level
A natural benchmark for the proposed limit (16 in this problem setting is the estimated quantile
t, (D) inf {e: (e) 2 21
(18)
using the inverse propensity weighted edf-estimator 110).
In all examples below, the limit 16 is computed using sample splits of equal size, i.c., no = [n/2]] An additional experiment using data from the Infant Health and Development Program (IHDP) can be found in Appendix A.2
The code used for the experiments is made available here https://github.com/SSt off Pi--policy--vlluatiio
5.1 Synthetic data
In the first example, consider synthetic data in order to evaluate the coverage of the derived limit curves. We use simulation setting similar to Jin et al. (2023). The miscoverage gap (I7 is estimated by Monte Carlo simulation using 1000 runs and for each run drawing independent 1000 new samples (Xn+1, Un+1, An+1, Ln+1).
We consider a population of individuals with two-dimensional covariates distributed uniformly as
X1 X2
X
2((0
The actions are binary A E {0, 1} corresponding to 'not treat' and 'treat', respectively. We want to evaluate a deterministic larget policy, described by
pr (A = -|x) 1X, 2 7)
(19)
for different 7 E [0, 1]. That is, all individuals whose covariate product X, falls below are treated. Note that 0 corresponds 'treat none' policy (A= 0 for all X) and 1 corresponds to 'treat all' policy (A = 1 for all X). Below we discuss the resulting losses under this policy using obscrvational data with sample sizes n E (250, 500, 1000].
Case: Known past policy (T' 1). In the first scenario, we assume that the past policy is known exactly and there no unmeasured confounding.
For the training data, the past policy has selected actions Bernoulli process:
oa== = p(4 = o =
CE
=
(20)
2

---

Published in Transactions on Machine Learning Research (06/2023)
where f()) is the sigmoid function. The conditional loss distribution is given by
(L = 0, ~ N11 0.11 and (L 1, X) ((11X,,001),
and thus Lmax We consider three configurations of past policies 20 which yield inverse propensity weights in three ranges: p1(A]] < 3.72 (c= 1/2), p((A 8.39 (c = 1), and P=(A[x] < 55.6 (c =c=33 2). Thus we anticipate Ps(A to be the most challenging case.
{0, 0.5, 1} in 19 and present their resulting limit curves using data from different past policies (20), see Figure 3 The dashed line shows the corresponding past policy, The limit curves for a given target policy are quite similar across training distributions and are still informative at the 90% level using The main difference is in the inferred tail losses and is notable for when 7 = 1 under the more challenging past policy Pa( (x)
Here evaluate three target policies
We now turn the evaluating miscoverage gap 19h. The solid lines illustrate the proposed method and the dashed lines show the benchmark (18). We see that the proposed method is slightly conservative, but remains valid for all a. In contrast, the benchmark is not valid in the tail of the distribution, but is less conservative for higher in this well-specified case.
Figure presents the gaps for target policy T 0.5
P)(A.))
Ps(AX)
80%
80%
80%
60%
60%
00%
40%
40%
40%
0.0
096
0%
0%
0.0
0.5
0.5
6
Figure 3: Limit curves when the past policy known (T = 1) for three different potential target policies
(i.e. {0.0, 0.5, 1.03 in 19 Dashed curve denotes the past policy. 1000.
250
0.150
0.150
0.155
0.125
0.125
11.125
Miscoverage gap
0.007
0.075
01,075
Post policy
0.050
0.000
0.000
0.025
pslA(X)
0.000
0.000
11.0001
Type
Progoscd
-0.025
-0.025
-0.025
Bunchmark
0.2
0.6
0.0
0.6
1.0
0.8
Target
Figure 4: Miscoverage gaps benchmark (18).
17 versus a, when the past policy is known (T 1). Dashod curve denotes the
1). In the second scenario, we assume that we only have access to an estimate X[ (given by (20 and that there is unmeasured confounding in the training distribution. To render visually distinct curves from the previous case, consider here rather extreme case of confounding following Jin et al. (2023).
Case: Estimated past policy (T >
Specifically, we have an unobserved variable drawn as
(UIX) ~ N(0,0.XX X2)),
8

---

Published in Transactions on Machine Learning Research (06/2023)
and the loss (L]A.X.U) is generated
1 1 X +U, A=
==
X1X2 ==
We define the past policy in manner that enables to control the divergence from the nominal model P(A[X) in 20
p = =|x=0] 1(U ((x)) 10A -p))= - -1)
(21)
+ 1U > +(X)) [I+Io -1
where the threshold function ((X) is designed empirically to ensure that the resulting median loss of the past policy for A = 1 is maximized. Our design of the past policy can be seen as the worst case among all unknown past policies that diverge by factor To in (6) We fix To here, but treat it as unknown.
For simplicity, we consider a 'treat all' target policy divergence bounds I' = {1, 2, 2), presented in Figure 5 Note that in the case of unmeasured confounding, the resulting curves differ notably across the training distributions unlike in Figure 3 We see that for the first and second distributions, the iveness of all curves stays around the 90% level. However, in the more extreme third case, the informativeness drops to just above the 60% level when we increase the credibility of our model assumption to an odds bound of 3. This example illustrates an inherent trade-off between credibility and informativeness.
1). Tts limit curves, under different assumed odds
guarantees using data drawn from (A[ When I 2 T 2, the limit curves are valid and as I' increases to 3, the limits become quite conservative. Note that the conservativeness persists even as the sample size n is increased fourfold. For r = 1, there is no guarantee of coverage and in this case the limit curve is invalid. The benchmark does not take the unmeasured confounding into account and is consoquently invalid throughout.
Figure 6 validates
p)(A(x)
p=)A)))
P=(A
80%
80%
80%
60%
60%
80%
40%
60%
40%
20%
20%
20%
0.0
1.0
6
Figure 5: Limit curves for 'treat all' larget policy using odds bounds I ={1,,,3), when the past policy is unknown and subject to unmeasured confounding (To 2 in (1 1000.
5.2
Real data
data from the National Health and Nutrition Examination Survey (NHANES) for the years 2013-2014 to illustrate the use of the proposed method. Following Zhao et al. (2019) study the effect of scafood consumption on blood mercury levels. The action A indicates whether a person has a low or high consumption of fish or shellfish (<1 vs. >12 servings in the past month) and the loss L is the total concentration of blood mercury (in pg/L). There are 8 covariates in X for each person: gender, age, income, whether income missing or not, race, oducation, ever smoked and number of cigarcttes smoked last month. 1 individual with missing education data and 7 with missing smoking data are omitted. 175 individuals have missing income data, for them impute the median income. This preprocessing results
In the second example,

---

Published in Transactions on Machine Learning Research (06/2023)
0.2
11.2
0.2
gap
-0.1
0.1
-0.1
Type
-02
-02
Proposed
Benchmakk
0.
0.2
0.4
0.0
0.8
0.0
0.2
0.4
0.6
0.5
0.0
0.2
0.4
0.6
0.8
1.0
Target
Figure 6: Miscoverage gaps 17 versus when the past policy is unknown and subject to unmeasured confounding. Dashed curve denotes the benchmark 18 which does not take confounding into account. Note that = assumes there is no odds divergence for our model, which in this case leads to an invalid limit curve.
in a data set of 1107 individuals of which 21% follow a high fish consumption diet and 79% follow a low consumption dict, see Zhao al. (2018). We use a fitted logistic regression model for ()) where the continuous covariates in X have been slandardized.
We compare two policies, тo and IT, corresponding to low (A = 0) and high (A 1) seafood consumption, respectively. Their corresponding limit curves are presented in Figure 7 under different odds divergences bounds I' on the model X)) In the most extreme case, we consider the nominal odds diverging by at most a factor r = 3. For reference, we display the limit curve under the unknown consumption policy in the population. We see that lower mercury levels can be certified by low seafood consumption. Note that guidance value for mercury levels is pg/L for women of thild-bearing age and μg/L for women 50 years (Kales & Thompson 2016). If high scafood consumption for women has expected loss of approximately 3.3 ug/ and could be deemed safe. However, this figure masks the lail losses that substantial proportion of women may face under such policy. For a high consumption policy, our method could only certify that approximately 80% (T = 1) to 50% (T = 3) females would stay below the guideline level of 8 ug/L. In contrast, for the low consumption policy, the corresponding figure is approximately 95% (for S r T 33)
to evaluate policies based the expected loss alone, then
All
Women
100%
100%
80%
80%
60%
30
40%
40%
Gamma
20%
20%
0%
0%
0.0
2.5
5.0
7.5
10.0
12.5
15.0
0.0
2.5
5.0
10.0
12.5
15.0
E, [ug/L]
[gg/L]
Figure 7: Limit curves for blood mereury levels population under different seafood consumption policies. Left: Overall population. Right: Female population. We compare two policies: low (лτ) and high (#1) consumption. The credibility of the assumptions increases with the odds bound I {1, The past policy is indicated by dotted curves.
10

---

Published in Transactions on Machine Learning Research (06/2023)
6 Conclusion
We have considered the problem of off-policy evaluation, i.c., making valid inferences about target policy using past observational data obtained under dilferent decision process with possibly unknown policy, Using the marginal sensitivity model, we derived sample-splitting method that provides limit curves with finite-amplle coverage guarantces even under model misspccifications, including unmeasured confounding. The validity, informativeness, and conservativeness of the resulting limit curves were demonstrated in the numerical experiments.
Acknowledgments
We thank the anonymous reviewers from TMLR for their constructive feedback. This research partially supported by the Wallenberg AT, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation, and the Swedish Research Council under contract 2021-05022.
References
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 (4):962-2)))) 973, 2005.
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceed- ings of the 26th annual international conference machine learning, pp. 49 56, 2009.
Yash Chandak, Scott Nickum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 0227475-274900
2021.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 1097-1104, 2011.
Alexander D' D'Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasject Sekhon. Overlap in observational studies with high-dimensional covariates. Journal of Econometrics, 221(2):644 654, 2021.
The Infant Health and Development Program. Enhancing the outcomes of low-birth- premature infants. multisite, randomized trial. JAMA, 263 22:3035-42, 1990.
Jennifer Hill. Bayesian nonparametrie modeling for causal inference. Journal of Computational and
Graphical Statistics, 20(1):217 240, 2011.
Daniel G Horvitz and Donovan Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American stafistical Association, 47(260):663-668 685, 1952.
Audrey Huang, Liu Leqi, Zachary Lipton, and Kamyar Avizzadeneshelii Off-policy risk assessment in con- textual bandits. Advances in Neural Information Processing Systems, 34:23714-23726, 2021.
Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6), 2023.
Stefanos N Kales and Aaron MS Thompson. A young woman concerned about mercury. CMAJ, 188(2): 134, 2016.
Nathan Kallus. Balanced policy evaluation and learning. Advances in neural information processing systems, 31, 2018
Nathan Kallus and Angela Zhou. Mininax-optimal policy learning under unobserved confounding, Manage- ment Science, 67(5):2870 200000000 2021.
11

---

Published in Transactions on Machine Learning Research (06/2023)
Joseph DY Kang and Joseph L Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating population mean from incomplete data. Statistical science, 22(4):523- -395999,, 2007.
John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems, 20(1):96 1, 2007.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Jing Lei and Larry Wasserman. Distribution-fres prediction bands for non non-parametric regression. Journal of the Royal Statistical Society: Series (Statistical Methodology), 76(1):71-96, 2014.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-fres prodictive inference for regression. Journal of the American Statistical Association, 113(6233:1094411111, 2018.
Charles F Manski. Identification problems in the social sciences and everyday life. Southern Economic Journal, 70(1):11121) 21, 2003.
Charles Manski. Patient Care Under Uncertainty. Princeton University Press, 2019.
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evalua- tion for sequential decisions under unobserved confounding. Advances in Newral Information Processing Systems, 33:18819-18881 2020.
Michael Oberst, Fredrik Johansson, Dennis Wei, Tian Gao, Gabriel Brat, David Sontag, and Kush Varshney. Characterization of overlap in observational studies. In International Conference on Artificial Intelligence and Statistics, pp. 788 798. PMLR, 2020.
Muhammad Osama, Dave Zachariah, and Peter Stoica. Learning robust decision policies from observational data. Advances in Neural Information Processing Systems, 1..18205-18214 2020.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Min Qian and Susan A Murphy. Performance guarantces for individualized treatment rules. Annals of statistics, 39(2):1180, 2011.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Paul R. Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41 55, 1983.
Andrea Rotnitzky, Quanhong Lei, Mariela Sued, and James M Robins. Improved double-robust estimation in missing data and causal inference models. Biometrika, 99(2):439 456, 2012.
Donald B Rubin. Using propensity scores to help design observational studies: application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2(3):169 188, 2001.
Joseph Schafer and Joseph Kang. Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychological methods, 13(4):279, 2008
Henry Scheffe and John W Tukey. Non-paramctric estimation. i. validation of order statistics. The Annals of Mathemalical Statistics, 16(2):187 192, 1945.
Glenn Shafer and Vladimir Vovk. A tutorial conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit exploration data. Advances in neural information processing systems, 23, 2010.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 10164)):1619-1633 2006.
Muhammad Faaiz Taufiq, Jean-Francois Ton, Rob Cornish, Yee Whye Teh, and Arnaud Doucet. Con- formal off-policy prediction in contextual bandits. In Alice Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho (ods.), Advances in Newral Information Processing Systems, 2022. URL
https://openrevieew www.:/oorunndddd1gg0000000 http:s//pppauuoooo..oocccoooooap
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction un- der covariate shift.. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Anastasios A Tsiatis, Maric Davidian, Shannon T Holloway, and Eric B Laber. Dynamic treatment regimes: Statistical methods for precision medicine. Chapman and Hal//CRC, 2019.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asion conference on machine learning, pp. 475-490. PMLR, 2012.
Vladimir Vork, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a rondom world. Springer Science Business Media, 2005.
Abraham Wald. An extension of wilks' method for setting tolerance limits. The Annals of Mathematical Statistics, 14(1) 14(1):45-56) 55, 1943.
Lan Wang, Yu Zhou, Rui Song, and Ben Sherwood. Quantile-optimal treatment regimes. Journal of the American Statistical Association, 113(523)):24331224 2018.
Daniel Westreich. Epidemiology by Design: A Causal Approach to the Health Sciences. Oxford Uni- versity Press, Incorporated, 2019. ISBN 9780190665760. URL https://books.ggtg se/books?id=
SR2yDwAAQBAJ.
Samuel S Wilks. Determination of sample sizes for setting tolerance limits. The Annals of Mathematical Statistics, 12(1):911900 96, 1941.
Baqun Zhang, Anastasios Tsiatis, Marie Davidian, Min Zhang, and Eric Laber. Estimating optimal trcatment regimes from a classification perspective. Stat, 1(1):103-114, 2012.
Qingyuan Zhao, Dylan S Small, and Paul R Rosenbaum. Cross-scroening in observational studies that test many hypotheses. Journal of the American Statistical Association, 113(523))1000--1884 2018.
Qingyuan Zhao, Dylan Small, and Bhaswar Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Slatistical Sociely: Series B (Statistical Methodology), 81(4):735 761, 2019.
Yingqi Zhao, Donglin Zeng, A John Rush and Michael R Kosorok, Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 107(499):1106 1118, 2012.
A Appendix
A.1 Proof of Theorem 41
The first part of the proof builds on techniques used derive weighted conformal prediction intervals Tibshirani al. (2019).
Let us consider sequence of no samples drawn from 4 followed by new sample drawn from 3 i.e.,
D+ ((xnot) Ungt1 Anoti; Img+1), (hott)..... ... (Xn, Un, An. Im), (Xat1, Un+1, Anti, Im-1))).
13

---

Published in Transactions on Machine Learning Research (06/2023)
The joint distribution of this sequence can be expressed using:
II 1=
pla,, uu, ai (i) p(αn+1, un An+1 kn+1) p(D+)wn+ = pp(+ (1S+)wn+1+
(22)
where n4 no + for notational simplicity, S+ is an unordered set of elements from D+, and the weight
Pulwi Wes ai Es)
04 p(r;, Uy Os, 0s,0,) 8.)
is the (unobservable) ratio 5 that quantifies the distribution shift from training to target distribution. We shall use the expression for the joint distribution to derive the distribution function for the new loss Ln+1-
Suppose we are given the unordered set S+ alone, then the particular sequence D+ is unknown. Let E, denote the event that the sample (X++1,nn+ Aat1, In)+) equals the ith sample (2t, u4 ai (,) the unknown sequence 4.. We consider all possible sequences D4 obtained by permutations a of elements in the set S+. Using the joint distribution in (22). the joint probability of the event Eg and S+ is then
S+)
Σ a=a(++1)==-)
p(S)), p(S+)w;n!,
where we have used the symbol P for the samples that are drawn from two different processes (4] and 3
The conditional probability of event Eg can now expressed as
P(E,S Ej P[E,,S}
Wi
P4 P[E[S+]
>n+1 W/
where the first equality follows from the law of total probability. The probability that the loss Ln+1 of the new sample equals l,, when conditioning the unordered set S4, is equal to
P(La+1 4|S+) = P[E[S+} = pi.
Thus conditional on St, the new loss Lnt1 has the following edf:
Enth w; 1 (Li 5)
P(Ln+1 S1S++ Σ pd14 5
(23)
a+t Zinn Wi
consider limit E that is function of the obscrvable elements in S4. For this part, we will build on the proof technique in (Jin et al. 2023 thm. 2.2).
Before marginalizing out S+ from 23
Specifically, using
13 we define the following limit:
1-
ta (D,, W w111 innf inf
(24)
for any 0 < 8 < a, where W n+1 Wa41 is given in 12 use the law of total expectation to marginalize out S.:
Now insert the limit (Di, W x+1) into 23 and
PP S (Di, W. -=11)]] = R. (P Tn & (D,, W [.+1)
W,ILL, (D), W w++1))
E
a+
Lin W:
We now proceed lower bound this probability. Note that by construction:
E |(la: D1, W Wnti)]
EiED, W, [T4 S.a)
(1aa
= =E EieD W,1(L, < <la) +EED, W,1(L, La) + W n+1
14

---

Published in Transactions on Machine Learning Research (06/2023)
Using this expression, we have that
(1 (k)
P( & la (D(,1, W ++1))
(1 - 3)
Zizny W (I, S la)
Eimn 1 E(a)
2 E
E
> jinn
W
Zieny W,1(L, S (a) + Di W,I(L: ba) + Wni1
E
Eatl Wi
Eien. W, S (,) + Ei=ny W,(L4 La) + W
where
"
Σ W.1LL, S Ska)
Σ
Σ
Σ W,1 Stao
Σ Wr
W.
Li=n
1=14
WL S L(a)
Σ
(K,1(, + Σ W,1LL, > xa) +
Σ WI (tao
Σ W.
iin
1=1+
2
Σ
WiL(L; s la)
Σ (2,(
(2) m+1
Σ W, S sla)
Σ
W.I(L, > (a) + W
1=
+i=n
+=
i=
2
Σ
WLL(L, s (.)
[
[
Σ
( - Σ W.1(L, &ca)
" Σ
W.1(L, > La) + Wn+1
imnj
iin
Limn
0,
using the bounds in to get the third inoquality. Therefore we obtain valid limit:
(-a)
PfIn+1 (D W w1))
(25)
(1
However, W n depends on a new sample drawn from the training distribution and thus the limit is unusable.
In lieu of W n+1, we use (D())) in 15 to define the modified limit
(a = inf {e: :: F(E;Di, 2
1-a
(26)
1-35
Comparing it with 24 we see that
t. (D) 2 (D,,,, Wnt1),
(27)
whenever W Wn-1 S wa (Da). By the construction in 15 the probability of this event lower bounded by
P(W n+1 S o (Do)) 21-
(28)
see Vovk et al. (2055) Lei et al. (2018)
We use this property to lower bound the probability of Ln+1 S (D))). First, note that
P(Inti S (a( W n+1 S w (Dn)) P(W n+1 S (D()) P(Inti S (D()
+P(LL+1 S (D( W n+1 > T (Do)) P W n-1 T (Po)) 2 P(In+1 $ (D(D) W n+ & "h (Da) P n+1 $ (Dn)) + 0.
15

---

Published in Transactions on Machine Learning Research (06/2023)
The first factor can be lower bounded using 27 that
P(Ln+1 S (D(D)) 2 F (D1, W x+1) W n+1 S (D P(W n+1 (
= L S (Di, W W++1)] P(W n+1 S w (Do)}
2 (1 a)
(29)
P(W n+1 S - (Dn)
(1 3)
2 1 - a.
The second line follows from using sample splitting, which ensures that Tn+11 S (D(1 W n+1) and W n+1
@ (Du) are independent events. The third and fourth lincs follow from (25 and (28 respectivcly. Since
229 holds for any 0 < B < a, we choose 3 in (26) that yields the tightest limit, ef. 16 Note that PP here
represents the probability over samples drawn from both 44 and (3). Since Ln+1 is drawn from (3) we can
write
29 as in for notational clarity.
A.2 Semi-real data
The Infant Hcalth and Development Program (IHDP) investigated the impact of carly childhood interven- tions on the health of low birgh- and premature infants (Health & Program 1990). Hill (2011) used this study to assemble data set of 25 covariates X that measured various aspects of the children and their mothers such as birth weight, weeks born preterm, head circumference and age of mother, etc. These covariates were standardized to have a zero mean and unit standard deviation. The data set also includes whether each child received special medical care or not, denoted by action A. The original study randomised control study, but the data were imbalanced was removed. This unbalanced data set contains 747 children, including 139 treated and 608 control units.
non-random subset of the treated population
The outcome was the child's cognitive development score, which we reverse it to an underdevelopment score to treat it as a loss L. The outcome is generated synthetically and we use 'response surface A' from Hill 22011::
1|A00,x X,,), L|A= N 4),
The vector is discrete valued with elements drawn from 1,,1,2,3,4 4} with probabilities (0.5, 0.2, 0.15, 0.1, 0.05). We use fitted logistic regression model for p(A[X). The limit (16] is computed using sample splits where no 0.1n. Another 0.1n samples of the data is randomly used to evaluate the policies in Figure g
We compare Iwo policies: the child did not receive special medical care ao ('treat none") or the child received special medical care πı ('treat all'). The limit curves are presented in Figure under different odds divergence bounds r = (1, 1.5, 2} on the nominal model. When I = 1, the 'treat all' policy is the policy resulting in the lowest limit curve. When I' is increased to 2 the limit curves are closer to each other, but the 'treat none' policy is instead the policy with the lowest. limit curve.
Figure g validates our guarantee in 2 The miscoverage gap 17 is estimated using 1000 simulation runs where we draw a new vector & and a new underdevelopment score for each run. The proposed method is valid for all values of T, but is more conservative for the higher values. The benchmark does not take the unmeasured confounding into account and is invalid for the 'treat all' policy (right).
16

---

Published in Transactions on Machine Learning Research (06/2023)
100%
80%
60%
40%
Pr p
Gamma T
20%
1.0
1.5
2.0
0%
20
-10
0
10
Ea
Figure & Limit curves for synthetic underdevelopment score in population under different policies. We compare two policies: the child did not receive special medical care xo the child received special medical care The past policy is indicated by dotted curves (px p). The credibility of the assumptions increases with the odds bound r = (1,1.5,2)
0.30
0.30
0.25
0.25
0.20
0.20
Miscoverage gap
0.15
0.15
0.10
0.10
Gamma
1.0
0.05
0.00
0.00
2.0
Type
-0.05
0.05
Proposed
Benchmark
-0.10
0.10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Target
Target
Figure 9: Miscoverage gaps 17 versus a for data from the Infant Health and Development Program (IHDP) under two different policies: the child did not receive special medical care mo or the child received special medical म1 Dashed curve denotes the benchmark (18) which does not take confounding or modelling errors into account. The credibility of the assumptions increases with the odds bound I (1, 2).
17


Reviews:
[{"review_id": "EYjKUfs7UE", "reviewer": "Reviewer_N8A4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts:**\n\nThe paper addresses an important topic, and the idea of deriving limit curves, rather than bounds on expected policy value, is interesting. The method seems to be a useful contribution and appears to work well. There is a rigorous justification for the method (although I could use some clarification).\n\n**Weaknesses**\n\n**W1:** I did not come away from the paper with a solid understanding of the role of each piece of the method. It would be useful to introduce the each part of the method and the role that they play in isolation before combining them together. In this respect, the marginal sensitivity model is well-motivated, but the limiting technique involving $\\beta$ is far less clear. I would be interested to see what the method looks like when the propensity score is assumed to be known and well-specified, and to have a better understanding of how the method differs from IPW in that case. Perhaps this was covered by previous work, but if so, it would be useful to have a short description in this paper.\n\n**W2:** It seems that the assumptions of the marginal sensitivity model are used in this work to handle both statistical uncertainty in estimating a model and misspecification/unobserved confounding. The former seems to be both well-characterized by previous asymptotic and finite-sample concentration arguments, and not especially amenable to the marginal sensitivity model framing where all errors $\\hat p(A \\mid X)$ need to be bounded probability 1. By contrast, most statistical uncertainty bounds are presented wrt some probability $\\delta$. How do the authors suggest reasoning about statistical uncertainty when choosing $\\Gamma$? Should $\\Gamma$ be chosen differently at different sample sizes? This would run contrary to advice in other parts of the causal inference literature, where $\\Gamma$ is considered purely a population quantity, rather than also quantifying assumptions about finite-sample statistical uncertainty.\n\n**W3**: The authors claim that the method handles finite sample uncertainty, but it seems like there is still uncertainty that is not quantified here. Setting aside the uncertainty in estimating $\\hat p(A | X)$ discussed above, there is also uncertainty in estimating $\\hat F$ that does not seem to be discussed (for example, in the proof, IIUC, arguments are made wrt the expectation of $\\hat F$). Could the authors clarify that this paper provides, essentially, a point estimate of the limit curves, but that there is additional statistical uncertainty in these limit curves? Relatedly, could uniform confidence bounds on CDFs, like the DKW bounds, be applied to obtain inference about the limit curves?\n\n**W4**: The proof could be made clearer, especially regarding the probability measures with respect to which the probabilities of events and expectations are being defined. See some of the minor comments below.\n\n**Minor Comments**\n\nThe role of $\\beta$ in Theorem 4.1 is not clear from the statement of the theorem. Could you provide some more intuition about it?\n\nPage 5: Could (12) be called something other than a “proxy” for the CDF? It’s essentially a lower-confidence bound on the CDF, correct?\n\nWhat is the interpretation of $\\beta$ in Theorem 4.1?\n\nOn page 6 it is claimed that $p(\\mathcal D_+)w_{n+1} = p(\\mathcal S_+)w_{n+1}$. This doesn’t seem right to me. Where does the combinatorial factor corresponding to permutations go?\n\nI’m a little confused about the probability measures wrt which many of the event probabilities are defined in Theorem 4.1. For example, in the display before (16), is the probability measure $\\mathbb P\\{L_{n+1} = \\ell_i \\mid \\mathcal S_+\\}$ defined with respect to $p$ or $p_\\pi$? It seems like it should be $\\mathbb P_\\pi$? Likewise, should the probability in the the event in (16) be wrt $\\mathbb P_\\pi$ not $\\mathbb P$? After eq (17), there is a $\\mathbb P_\\pi$ inside they expectation, but what measure is the outer expectation taken over?\n\nShould the limit in (17) be a function of both $\\alpha$ and $\\beta$? I.e., should it be written $\\ell_{\\alpha, \\beta}(\\mathcal D_1, \\bar W_{n+1})$?\n\nIt seems strange in (28) to define the ground truth as a function of the estimated nominal propensity score. It would seem more realistic to define the estimation procedure such that $\\hat p(A =0 | x)$ concentrates appropriately to satisfy the $\\gamma$ bound with high probability.\n\nThere should probably be a sentence that says “informativeness” corresponds to the right limit of the curves in Figures 3, 5 (I think). It was difficult for me to interpret at first.\n\nIt would be useful to have a clear description of how the proposed method differs from the IPW method in the correctly specified case.\n\nIt would be easier to interpret the takeaway from the NHANES experiment if the high-consumption rate of shellfish under the current policy were also given. I.e., if high consumption is relatively rare, then it makes sense that the rate of exceeding the guideline level will be relatively low (closer to $\\pi_0$) under the current policy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for building prediction bounds on out-of-sample loss when a new policy is applied to a given population. The method aims to provide finite-sample guarantees even under model misspecification and unobserved confounding, under assumptions of varying strength under a marginal sensitivity model. The method makes use of a conformal argument combined with data splitting, and a proof is given that combines elements from two previous works on conformal arguments under distributon shift, and on bounding ITEs under confounding. The method is shown to be valid in two simulated settings; one with a well-specified, known propensity model, and one with unobserved confounding and an estimated propensity model. The method is then demonstrated on NHANES data regarding mercury levels and shellfish consumption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are copied from above:\n\n - A description of what the method does when $\\Gamma = 1$, with a comparison to the IPW baseline. This would give some intuition about the core approach before we insert bounds on importance weights.\n - Some intuition about the quantities $\\beta$ and $\\bar w_\\beta(\\mathcal D_0)$, and what their role is.\n - Some commentary, and refinement of claims, regarding finite sample statistical uncertainty, in both the estimation of $\\hat p(A | X)$ and $\\hat F$.\n - Clarifications to other minor points, including double-checking notation in the proof.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BEudap6f5k", "reviewer": "Reviewer_s42E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe introduction of the problem and motivation is clear. The problem of study is relevant.\n\nMeasuring the entire loss distribution seems novel to me, since most existing results are for expected values.\n\nThe proposed methods are with guarantees and supported by experimental results.\n\n**Weaknesses**\n\nThe usefulness of the new metric (entire loss distribution) in off-policy evaluation is not well explained.\n\nThe authors do not provide explanations for the proposed methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of off-policy evaluation, i.e., evaluating the performance of a decision policy using past data. The authors provide results for the entire loss distribution instead of the expected value of policies, i.e., Eq. (2).\n\nSection 3 summarizes existing baseline methods. There exist estimators of the mean, as noted in Eqs. (7) and (8), and the doubly robust estimator, and similar estimators can be generalized to estimate entire loss distributions, as shown in and after Eq. (9).\n\nSection 4 introduces the methods based on splitting the dataset and the proposed estimator in Eq. (12), and presents the main results, i.e., the limit in Eq. (15) satisfies Eq. (2). The authors then give the implementation (Algorithm 1) and prove Theorem 4.1.\n\nSection 5 contains experiments. The authors first define several metrics, including informativeness (Eq. (23)), miscoverage gap (Eq. (24)). They use synthetic data and real data from the the National Health and Nutrition Examination Survey (NHANES). The results show that the informativeness of most curves stays a high level (about 90%), and the proposed method is slightly conservative, but remains valid for all $\\alpha$ values.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "First, I get that the metric of entire loss of distribution is different with the expected loss. However, I did not get how the entire loss of distribution is useful in off-policy evaluation. The authors mention that in the introduction, when the distribution of losses is skewed or is widely dispersed, expected loss itself is limited. I am wondering show the entire loss distribution could be used after off-policy evaluation. Some concrete examples of showing that the entire loss distribution could provide a different solution that using expected loss would be great.\n\nSecond, the baseline methods are well discussed in Section 3, since they look closely related to existing mean estimators. However, in Section 4, the proposed method is presented without much intuitions or discussion. The authors mention that in Section 3 \"this estimator is\nconsistent, it is not guaranteed to yield a proper cdf\". I would like to see more comments on why the authors design the proposed estimator in such a way and how intuitively the proposed methods could resolve issues of existing methods.\n\nThird, the proposed method does not change over different values of $\\alpha$ in Figures 4 and 6. Could you comment on this.", "broader_impact_concerns": "The experiments are conduced using synthetic data and public dataset. I do not see concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ehpYkxkiF", "reviewer": "Reviewer_jTUo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The split method is easy to implement, and alleviates the strong assumption which is required by previous methods. \n\n- As stated previously, the assumption here is relatively mild, which is good for real-world applications. \n\n- The proof written in the paper is easy to follow. \n\n\n**Weaknesses**\n\n- There is not much baselines to compare in the experiment part, and also I haven't see any discussion with previous work or standard approaches.\n\n- Still, the experiments part is a little bit weak. There is not much more real experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the problem of off-policy evaluation, where we want to evaluate the performance of a target policy with only past observational data. The authors leverage recent advances in conformal predictions to obtain finite-sample coverage guarantees over the entire loss distribution. Due to the benefits of conformal prediction techniques, the new approach allows to take model misspecifications into account. The authors validate their methods on both synthetic data and real world dataset, and can obtain relatively reasonable coverage and out-of-sample guarantees. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add a discussion with your method and some classical approaches such as utilizing confidence bounds to obtain confidence intervals, or some distributional robust optimization based approach (e.g., Empirical likelihood for contextual bandits).\n\n- Please add some more real-world experiments. \n\n- Since your method requires data spliting, can you comment or discuss the advantage and disadvantage of splitting based methods under the OPE setting? e.g., you can not leverage all the samples to construct the upper and lower bound? \n\n\n- Please add a related work session and move the proof part to the appendix. \n\n- please distinguish citet and citep. \n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dm5T1Jmt6u", "reviewer": "Reviewer_ggu1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1. An important problem that might be of high relevance to the audience\n\nS2. Clearly written and to the point. Although I would have liked more discussions on assumptions, intuitions about the formulation, and clarifications (as mentioned below)\n\nW1. I feel the motivation and the problem statement might be a bit disjoint.\n\nW2. Technical novelty is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at the problem of off-policy evaluation and inference, where the observed data might be subject to unobserved confounding. The key assumption being made to get around this issue of unobserved confounding is to assume a known bound on the odds ratio (Eqn 6). Under this assumption, the authors theoretically show how techniques from conformal inference can be used to get coverage over the entire distribution of loss/outcomes. An empirical investigation of a toy and a real-world example is provided to support the claim.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Worth double check the use of phrase `external validity'. I have mostly seen the causal inference community uses that term for distribution mismatch of the covariates (distribution of X) or outcome (reward/cost/loss) models but the treatment policy is held constant. The setting being considered in the paper is where distribution of covariates and reward function is the same, but the treatment policy is different.\n\n2. \" loss no greater than ℓα(D) under policy π [with high confidence].\"\n\n3. Eqn 6 needs to made more precise. Is the ratio the worst case ratio over all possible X-U-A triplet?\n\n4. Other related work around eqn 6 in the OPE literature (although for the mean, and not for quantile/CDF):\n\t\nA Namkoong, Hongseok, et al. \"Off-policy policy evaluation for sequential decisions under unobserved confounding.\" Advances in Neural Information Processing Systems\n\t\nB Giguere, Stephen, et al. \"Fairness guarantees under demographic shift.\" International Conference on Learning Representations. 2022.\n\t\n5. Is the \"nominal model\" $\\hat p(A|X)$ the probability of action given covariates estimated from the data, or is it the true value $\\int_u p(A|X,u) du$? Interpretation of bounds in Eqn 6 will be different based on this, and this should be properly clarified.\n\n6. I think Eqn 2 and problem statement needs a more nuanced discussion. If I understand it right, the citations for Chandak et al. and Huang et al. provide PAC-like (1-\\epsilon accurate with 1-\\delta confidence) coverage, i.e.,  $Pr_D(Pr_X(X < fn(D)) > 1- \\epsilon) > 1-\\delta$. The coverage in Eqn 2 is slightly different (and a weaker guarantee) as both $L_{k+1}$ and $D$ are random variables.   This is not to say that Eqn 2 is bad, but needs to be properly justified to the reader for why it is meaningful towards the motivation that preceded it.\n\nSee discussion around Eqn 5 (Vovk 2012), Eqn 1 and 2 (Danijel et al. 2020), and others (Sangdon et al.)\n\nVovk, Vladimir. \"Conditional validity of inductive conformal predictors.\" Asian conference on machine learning. PMLR,\n\nKivaranovic, Danijel, Kory D. Johnson, and Hannes Leeb. \"Adaptive, distribution-free prediction intervals for deep networks.\" International Conference on Artificial Intelligence and Statistics.\n\nPark, Sangdon, et al. \"PAC prediction sets under covariate shift.\" arXiv preprint arXiv:2106.09848 (2021).\n\n7. While $\\bar w$ is being estimated through data splitting, wouldn't the optimization process is Eqn 15 make the choice of $\\bar w$ indirectly depend on both the partitions of the data? What is the intuition for introducing $\\beta$ and what is the trade-off that $beta$ is balancing, e.g., why is $\\beta=\\alpha$ not adequate?\n\n8. An important factor that separates this work from prior methods (Chandak et al. (2021) and Huang et al. (20221) is that it can deal with settings where the observational data has unobseved confounding, and this is possible by controlling the odds ratio (Eqn 6).\n\n Is there any particular reason to consider the ratio of the odds in Eqn 6 instead of bounding $p_\\pi(A|X)/p(A|X,U)$ directly? Empirically, it _seems_ like a practitioner will have a better idea of the latter. Theoretically, could one not use these bounds in the works by Chandak et al. (2021) and Huang et al. (20221) to directly extend their bounds for the desired setting with unobserved confounders and unknown behavior policy? It feels like this would just affect the leading constant on the Hoeffding/DKW-like bound which currently depends on the max of the IPW/importance ratio's values.\n\n9. I only got a chance to skim through the proof but I think as a reader it would be helpful if the authors can discuss intuitively what Eqn 13 and 15 are capturing as well. Here it will also be beneficial to rediscuss the setting where $\\tau=1$, similar to how it was done in Section 3. These might make the draft more readable and perhaps the proof can be deferred to the appendix.\n\n10. I am intrigued by the non-monotonous nature of the curves in Figure 4 and 6. Can authors share insight on why it is not monotonous?\n\n11. Can authors discuss how to leverage their proposed method when one only requires to get coverage over the expected value of the loss/reward for the evaluation policy? (I suspect this might not be possible and would connect back to point 6 discussed earlier. But maybe I am wrong, and perhaps authors can correct me over here. )\n", "broader_impact_concerns": "Since this paper deals with topics that aim at safety and credibility, it might benefit from more discussion on assumption implied by Eqn 6 on the bounds of the odds ratio. How should this be chosen in practice, etc.?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the reviewers' recommendations, the paper is worth publication in TMLR. However, one of the reviewers asked for clarification on the primary sensitivity parameter introduced by the authors. Before acceptance, the authors should address their concerns or at least add some details to the paper. 

Decision: Accept (poster)