Conformal Off-Policy Prediction in Contextual Bandits
Authors: Muhammad Faaiz Taufiq
Jean-Francois Ton
Rob Cornish
Yee Whye Teh
Arnaud Doucet
Venue: NeurIPS 2022 Accept (2022)
Keywords: conformal prediction
contextual bandits
uncertainty quantification
robust ML

Abstract: Most off-policy evaluation methods for contextual bandits have focused on the expected outcome of a policy, which is estimated via methods that at best provide only asymptotic guarantees. However, in many applications, the expectation may not be the best measure of performance as it does not capture the variability of the outcome. In addition, particularly in safety-critical settings, stronger guarantees than asymptotic correctness may be required. To address these limitations, we consider a novel application of conformal prediction to contextual bandits. Given data collected under a behavioral policy, we propose \emph{conformal off-policy prediction} (COPP), which can output reliable predictive intervals for the outcome under a new target policy. We provide theoretical finite-sample guarantees without making any additional assumptions beyond the standard contextual bandit setup, and empirically demonstrate the utility of COPP compared with existing methods on synthetic and real-world data.

Conformal Off-Policy Prediction in Contextual Bandits
Muhammad Faaiz T Department of Statistics University of Oxford
Jean-Francois Ton* Al-Lab-Research Bytedance Al Lab
Rob Cornish
Yee Whye Teh Department of Statistics University of Oxford
Arnaud Doucet Department of Statistics University of Oxford
Department Statistics University of Oxford
Abstract
Most off-policy evaluation methods for contextual bandits have focused the expected outcome of a policy, which is estimated via methods that at best provide only asymptotic guarantees. However, many applications, the expectation may not be the best measure of performance as it does not capture the variability of the outcome. addition, particularly in safety-critic critical settings, stronger guarantees than asymptotic correctness may be required. To address these limitations, we consider novel application of conformal prediction contextual bandits. Given data collected under behavioral policy, we propose conformal off-poolicy prediction (COPP), which can output reliable predictive intervals for the outcome under a new target policy. We provide theoretical finite-sample guarantees without making any additional assumptions beyond the standard contextual bandit setup, and empirically demonstrate the utility of COPP compared with existing methods on synthetic and real-world data.
Introduction
Before deploying a decision- makinn policy to production, is usually important understand the plausible range of outcomes that it may produce. However, due to resource or ethical constraints, it is often not possible to obtain this understanding by testing the policy directly the real-world. such cases we have to rely on observational data collected under different policy than the target. Using this observational data to evaluate the target policy known as off-policy evaluation (OPE).
Traditionally, most techniques for OPE in contextual bandits focus on evaluating policies based However, this can be problematic as methods that are only concerned with the average outcome do not take into account any notions of variance, for example. Therefore, in siskisensie settings such as econometrics, where we want to minimize the potential risks, metrics such as CVaR (Conditional Value at Risk) might be more appropriate [ Additionally, when only small sample sizes of observational data are available, the average outcomes under finite data can be misleading, they are prone to outliers and hence, metrics such as medians or quantiles are more robust in these scenarios 1.
on their expected outcomes; see e.g., 122 30
26
2
24
Instead of estimating bounds on the expected outcomes, [ 55) establish finite-sample bounds for general class of metrics (c.g., Mean, CVaR, CDF) on the outcome. Their methods can be used to estimate quantiles of the outcomes under the target policy and are therefore robust to outliers. However, the resulting bounds do not depend on the
Notable exceptions in the OPE literature are
DDnotees equal contribution, where ordering was determined through coin flip. Corresponding authors muhammad. tauf i gestats. ox. ax.ac. uk and jeanfrancois@byt com.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Observational Data
Goal: Predict recommendation relevance under target policy
π
Traditoone OPE
COPP
Out
COFF Dors
Bigel
Averaage infexunce rale:
Jolrord
Figure 1: Left (a): Conformal Off-Policy Prediction against standard off-policy evaluation methods. Right (b): 90% predictive intervals for Y against X for COPP, competing methods and the oracle.
covariates X (not adaptive w.r.t. X). This can lead to overly conservative intervals, will show in our experiments and can become uninformative when the data are heteroscedastic (see Fig. (
In this paper, we propose Conformal Off-Policy Prediction (COPP), novel algorithm that uses Conformal Prediction (CP) 22 to construct predictive interval/sets for outcomes contextual bandits (see Fig T using an observational dataset. COPP enjoys both finite-sample theoretical guarantees and adaptivity w.r.t. the covariates X, and, to the best of our knowledge, the first such method based on CP that can be applied to stochastic policies and continuous action spaces.
In summary, our contributions are: (i) We propose an application of CP to construct predictive intervals for bandit outcomes that is more general (applies to stochastic policies and continuous actions) than previous work. (ii) We provide theoretical guarantees for COPP, including finite-sample guarantees marginal coverage and asymptotic guarantees on conditional coverage. (iii) We show empirically that COPP outperforms standard methods in terms of coverage and predictive interval width when assessing new policies.
1.1 Problem Setup
Let X' be the covariate space (e.g., user data), A the action space (e.g., recommended items) and Y the outcome space (e.g., relevance to the user). We allow both A and y to be either discrete or continuous. In our setting, we are given logged observational data Dabs actions are sampled from a behavioural policy n², i.e. A We assume that we do not suffer from unmeasured confounding. At test time, we are given a state gteat and a new policy π²" While x² may be unknown, we assume the target policy x' be known.
{ar, ag Yi Pabu where x- x)) and Y x.
P(: x, a).
We consider the task of rigorously quantifying the performance of nt without any distributional assumptions X or Y. Many existing approaches estimate E,- [Y], which useful for comparing two policies directly as they return a single number. However, the expectation does not convey fine-grained information about how the policy performs for specific value of X, does it account for the uncertainty in the outcome Y.
Here, we aim to construct intervals/sets the outcome Y which are (i) adaptive w.r.t. X, (ii) capture the variability in the outcome Y and (iii) provide finite-sample guarantees. Current methods lack at least one of these properties (see Sec. 5 One way to achieve these properties is to construct set-valued function of x, C(x) which outputs subset of y. Given any finite dataset Dots. this subset is guaranteed to contain the true value Y with any pre-specified probability, i.e.
a 1- ((XY)) (Y O(X))
Onose (1)
where nobs is the size of available observational data and PÆ, is the joint distribution of (X, Y) under target policy πτ. In practice, ((xx) can be used as diagnostic tool downstream for a granular assessment of likely outcomes under target policy. The probability in ( is taken over the joint distribution of (X, Y), meaning that (I) holds marginally in X (marginal coverage) and not for a given X (conditional coverage). In Sec. 4.2 we provide additional regularity conditions under which not only marginal but also conditional coverage holds. Next, we introduce the Conformal Prediction framework, which allows us to construct intervals ((z) that satisfy I along with properties (i)-(ii).

---

2 Background
Conformal prediction 29 19
is methodology that was originally used compute distribution-free data Pe- { Y iid.
prediction sets for regression and classification tasks. Before introducing COPP, which applies CP to contextual bandits, first illustrate how CP can be used in standard regression.
2.1 Standard Conformal Prediction
Consider the problem of regressing E y against X E X. Let f be model trained on the training
Px,y and let the calibration data Deat = {X,, Yi i.d. Рx,y be independent of Der. Given a desired coverage rate 1 - a E (0, 1), we construct a band Cn : X + {subsets of )) based on the calibration data such that, for i..d. test data (X Px,Y,
1 a - P(x (Y E (x)) s a+ n n+1
(2)
where the probability is taken over X, Y and Deat = (X,, Y. and is conditional upon Dtr-
In order to obtain On satisfying 2 we introduce non-conformity score function V; = s(,,, Yi). e.g.. (f(X,) - Y.)². We assume here (Vi) have no ties almost surely. Intuitively, the non- conformity score Vú uses the outputs of the predictive model ff on the calibration data, to measure how far off these predictions are from the ground truth response. Higher scores correspond worse fit between æ and y according to f. We define the empirical distribution of the scores (Vi)_= U (oc)
Fn = n+1 Σ Sv. + n+1 6o0
(3)
with which we can subsequently construct the conformal interval Ć, that satisfies Cn(x) tv s(x,y) S n)
2 as follows:
where 1) is an empirical quantile of (Vi) i.c. = Quantile,-a is the 1 - quantile.
Intuitively, for roughly 100 (1 a)% of the calibration data, the values will be below 1)- Therefore, the new datapoint (X, Y) and Deal are i.i.d., the probability PPs((X,)) 7) (which is equal to P(Y E (x)) by 4 will be roughly 1 - a. Exchangeability of the data is crucial for the above to hold. In the next section we will explain how [ relax the exchangeability assumption.
2.2 Conformal Prediction under covariate shift
P extend the CP framework beyond the setting of exchangeable data, by constructing valid intervals even when the calibration data and test data are not drawn from the same distribution. The authors focus on the covariate shift scenario i.e. the distribution the covariates changes at test time:
(X,, sid Px, = Px x Prix: i= i,.,..,
(X,Y) ~ Px, = Px x Pr|x, independently
where the ratio w(x) = dPx /dPx is known. The key realization in 27 is that the requirement of exchangeability in CP can be relaxed to more general property, namely weighted exchangeability (see Def. I. They propose a weighted version of conformal prediction, which shifts the empirical distribution of non-conformity scores, F.. at point æ, using weights w(x). This adjusts F, for the covariate shift, before picking the quantile 1y:
p (x)óv; + (x)00
where,
Ps (x)
(X.) Σ (xx)+ P (x) Σg w(X))+((x)
w(x)
In standard CP (without covariate shift), the weight function satisfies w(x) = for all i,, and we recover Next, we construct the conformal prediction intervals C, as in standard CP using where y) now depends on x due to p\ (x). The resulting intervals, On, satisfy:
P(x,Y)~Xx, (Y E (x)) 2
3

---

As mentioned previously in Sec. 1.I the above demonstrates marginal coverage guarantees over test point X and calibration dataset ent. not conditional on a given X will discuss this nuance later on in Sec. 4.2 In addition, previous work by Vovk (28 shows that conditioned on single calibration dataset, standard CP can achieve coverage that 'close' to the required coverage with high probability. However, this has not been extended to the case where the distribution shifts. This is out the scope of this paper and an interesting future direction.
fixed D Dent- We
Algorithm 1: Conformal Off-Policy Prediction (COPP)
Inputs: Observational data Dobe {Xx,, A., Yi conf. level a, score function s(x, y) E R, new data point
_ p target policy n²;
Output: Predictive interval Č₂ (z*"x));
Split Doba into training data (Dtr) and calibration data (Deai) of sizes m and n respectively; Use Der to estimate weights -)- using 7
Compute Vi = s(X,, Yi) for (Xs, A,, Yi) E Deal:
Let F be the weighted distribution of scores F Li=1 P." (a, y)vv, Ph+ (2,8)6 xY)
where p. (I, E ,X and ph4 (I, = Σラ ,
For test construct: Cn (zttns))= >= {y: s(x test y) S Quantile (F |)
Return (a
Thus show that the CP algorithm can be extended to the setting of covariate shift with the resulting predictive intervals satisfying the coverage guarantees when the weights are known. The extension of these results to approximate weights proposed in [4] and is generalized to
setting in Sec.
3 Conformal Off-Policy Prediction (COPP)
In the contextual bandits introduced in Sec. 1.1 we assume that the observational data Dote {z,, ass 3/4 Potk is generated from a behavioural policy x². At inference time we are given new target policy x² and want to provide intervals on the outcomes Y for covariates X that satisfy (
The key insight of approach is consider the following joint distribution of (X, Y):
p= (z, -P(x) / P(yla, (lllx,a)" ())" P(x) (y/z)
pr (x, -P(x) / Plule, a)r (al=)da = P(x) pr (yla)
Therefore, the change of policies from n to causes shift the joint distributions of (X, , Y) from X,y to Px,y- More precisely, shift in the conditional distribution of Y|X. As result, our problem boils down to using CP in the setting where the conditional distribution yyix changes to pr'
due to the different policies, while the covariate distribution Px remains the same. YIx
Hence our problem not concerned about covariate shift as addressed in (27], but instead uses the idea of weighted exchangeability to extend CP to the setting of policy shift. To account for this distributional mismatch, our method shifts the empirical distribution non-conformity scores at a point (x, y) using the weights w(x, y) = /dPx' (x, = xxpix//Pu y|x (x, y):
fr Σ ps (F (1,y) p (x. y))
(5)
m(x.g) pfd (Ix, y) 1o (u) P (2, ΣT w w (X, w(
(X.M)
The intervals are then constructed as below which we call Conformal Off-Policy Prediction (Alg. M
(zrest) = (v textest ,y) I festt pooeestt)) where, 171,, y) == Quantile, (fr.v). (FF..)
(6)

---

Remark, The weights w(z,y/ v) in 5 depend on r and y. as opposed to only x. In particular, finding the set of y's satisfying (6) becomes more complicated than for the standard covariate shifted CP which only requires a single computation of 7)(z)) for given we have to create k sized grid of potential values of for every to find (z). This operation is embarrassingly parallel and hence does not add much computational overhead compared to the standard CP, especially because CP mainly focuses scalar predictions.
shown in 4 In our case however,
3.1 Estimation of weights w(z, v)
So far have been assuming that we know the weights w(r, y) exactly. However, in most real-world settings, this will not be the case. Therefore, we must resort to estimating w(æ, y) using observational data. In order to do so, we first split the observational data into training (Dty) and calibration (Deat) data. Next, using Dtr. we estimate (a x) z (a z and P(y ar Ply xz a) (which is independent of the policy). We then compute Monte Carlo estimate of weights using the following:
w( # Ek= Plylar, AT)
[ P(ylaa a)_,,aax"" (a|x)da (P (a)a
T E Plulær A&)
where As ~ (- 1)), A ~ π (.1 x)) and is the number of Monte Carlo samples.
Why not construct intervals using P(ylæ, a) directly? We could directly construct predictive intervals (x) over outcomes by sampling Y, lid. p (g|z) = f P(ylæ, a) (a)"(a) However, the coverage of these intervals directly depends on the estimation error of P(y|zz, a). This is not the case in COPP, the coverage does not depend on P(y]ar, a) directly but rather the estimation of (0 y) (see Prop. 4.2 We hypothesize that this indirect dependence of COPP on P(y) æ. a) makes it less sensitive to the estimation eπrоr. In Sec. 6 our empirical results support this hypothesis as COPP provides more accurate coverage than directly using P(ylæx, a) to construct intervals. Lastly, in Appendix B.5.2 we show how we can avoid estimating P(y|z, a) by proposing alternative method for estimating the weights directly. We leave this for future work.
4 Theoretical Guarantees
4.1 Marginal Coverage
In this section we provide theoretical guarantees on marginal coverage ((,,))~ (Y E (x)) for the cases where the weights w(I, y) known exactly well when they are estimated. Using the idea of weighted exchangeability, we extend (27] Theorem 2] to our setting.
Proposition 4.1. Let (X,, Yi)==1 iid. Py'x be the calibration data. For any score function s. and any E 1). define the conformal predictive interval at point E Rd as C,(2)) ty R. s(x,z) S m(x,v))) y)} where n(I, v) == (aa and FENM is as defined in with exact weights w(x, y). If pr (y/z) is absolutely continuous w.r.t. p= (y)zz), then On satisfies F ((x.)>PP'' (Y E (X)) 2 1
Proposition 4.1 assumes exact weights w(x, y), which is usually not the case. For CP under covariate shift, [4] showed that even when the weights are approximated, i.e., ú(x y) o w(x, y), we can still provide Enite-sample upper and lower bounds on the coverage, albeit slightly modified with an error term A. (see KD). We show that this result can be extended to our setting when the weight function w(z, y) is approximated as in Sec. 3.1
Proposition 4.2. Let Ön be the conformal predictive intervals obtained as in Proposition 4.1 with weights w(z, y) replaced by approximate weights w(2,) v) úkz y: Drr), where the training data,
Der, fixed. Assume that rò u0. y) satisfies (E (E. [x, YYT) S M, < so for some
2 2. Define Aw as,
A w L E ((XYY) w(X,
(8)
Then, P Pix.rrgg (Y Cx())) > - AW.
5

---

If, in addition, non non-conformity scores [V:)== have no ties almost surely, then we also have
P ((XY)) (Y E (x)X) & - a + Aw + cn 1/-11
for some positive constant с depending only on M, and r.
Proposition 4.2 provides finite-sample guarantees with approximate weights u(.,)). Note that if the weights are known exactly then the above proposition can be simplified by setting Aw = 0. the case where the weight function estimated consistently, we recover the exact coverage asymptotically, A natural question to ask is whether the consistency of (z, y) implies the consistency of P(y|a: a); in which case one could use P(y|z, a) directly to construct the intervals. We prove that this not the case in general and provide detailed discussion in Appendix B.5
4.2 Conditional Coverage
So far we only considered marginal coverage I where the probability over both X and Y. Here, (Y E (X) X) which is strictly stronger prove that exact conditional coverage cannot be achieved without making additional assumptions. However,
we provide results on conditional coverage PT x
notion of coverage than marginal coverage X
show that, in the case where Y is a continuous random variable and we can estimate the quantiles of y consistently, we get an approximate conditional coverage guarantee using the below proposition.
Proposition 4.3 (Asymptotic Conditional Coverage). Let m, n be the number of training and calibra- tion data respectively, (..m(ar) ) = $3,m (1x; Dtr) be an estimate of the 3-th conditional quantile qs (x) of =ix== Wm (x, y) = Tm (2,, y; Dir) be an estimate of w(t, y) and mm (x) be the conformal
interval resulting from algorithm 7 with score function s(x, y) max(y (2)) Uaso (I) y}
where Ohii"
0:1o = 1 a. Assume that the following hold:
1. 1oo E (x,Y) (x,Y)~x'' im (X, Y) w(X, Y) -.
2. there exists r, bi, b2 O such that pa (y x)) E [b2. d2] uniformly over all (ar y) with y E 19 (x) r, Acmo (xr) + rr] 19as (x) - 9ans (z) + +r)
3. 3k > s.t. lomm- E XxxPxxx Px [ (X)]
where Hm(()) max (o
Jat (z),, Aons. (x) Jan. (x)
Then for any t O, we have that lim P(Pr (Y Cm.n (X) X) <
0. rather strong. In general, consistently
Remark. One caveat of Prop. 4.3 is that Assumption estimating the quantiles under the target policy n" is not straightforward given that we only have access to observational data from nb. While one can use a weighted pinball loss to estimate quantiles under x² consistent estimation of these quantiles would require consistent estimate of the weights (see Appendix C Hence, unlike [4 Theorem 1], our Prop. is not a "doubly robust" result.
Towards Group Balanced Coverage. As pointed out by 2 we may want predictive intervals that have the same coverage across different groups, e.g., across male and female users [17]. Standard CP will not necessarily achieve this, as the coverage guarantee ① is over the entire population of users. However, we can use COPP on each subgroup separately to obtain group balanced coverage. A more detailed discussion on how to construct such intervals has been included in Appendix [.4
Related Work
Conformal Prediction: A number of works have explored the use of CP under distribution shift. The works of (27 and [14] are particularly notable as they extend CP to the general setting of weighted exchangeability. In particular, 14| use CP for counterfactual inference where the goal is to obtain predictive intervals on the outcomes of treatment and control groups. The authors formulate the counterfactual setting into that of covariate shift in the input space X' and show that under certain assumptions, finite-sample coverage can be guaranteed.

---

Table 1:
Toy experiment results with required coverage 90%. While WIS intervals provide required coverage, the mean interval length is huge compared to COPP (see table lb
(a) Mean Coverage as function of policy shift with 2 standard errors over 10 runs.
(b) Mean Interval Length as function of policy shift with standard errors over 10 runs.
Coverage
0.0
A. 0.1
0.2
Interval Lengths
A. = 0.0
à. = 0.1
4. = 0.2
CDPP (Ours)
0.900
0.90
0.01
0.91
COPP (Ours) WIS SBA
9.08 + 0.10
9.48 + 0.22
9.7 + 0.38
WIS
0,89 :: 0.01
0.91
= 0.02
0.94
2 0.02
24.14 + 0.30
32.96 + 1.00
43.12 3.49
SBA
0.90
0.01
0.88
0.01
0.7
+ 0.01
8.78 4.78:
8,94 + 0.10
8,33 0:00
COPP (GT weights Ours)
0.90 1: 0.01
0,90
=: 0.01
0.90 1: 0.01
COPP (GT weights Ours)
8.91 1001+0000 0.00
9.25 + +.12
9.59 0.20
CP (no policy shift)
0.90 0.01
0.87 0.01
0.85 0.01
CP (no policy shift)
9.00.10.110
9.00
1 1,10
9.00 1.300
CP (union)
2: 0.01
0.96
0.01
0.96
0.01
CP (union)
10.66 0.188
11,04 1: 0.2
11.4 114:::0:0
Fundamentally, our work differs from [4] by framing the problem as a shift in the conditional Py rather than as a shift in the marginal Px. The resulting methodology we obtain from this then differs from [4]] in a variety of ways. For example, while [14] assume deterministic target policy, COPP can also applied to stochastic target policies, which have been used in a variety of applications, such recommendation systems or RL applications 25 22 Likewise, unlike [4], COPP is
applicable to continuous action spaces, c.g., doses of medication administered.
In addition, when the target policy is deterministic, there is an important methodological difference between COPP and [4] In particular, [4] construct the intervals on outcomes by splitting calibration data w.r.t. actions. In contrast, it can be shown that COPP uses the entire calibration data when constructing intervals on outcomes. This is a consequence of integrating out the actions in the weights w(x v) 7. and empirically leads to smaller variance in coverage compared to [4.. See B.1 for the experimental results comparing COPP to 14] for deterministic policies.
15 propose using CP to construct robust policies in contextual bandits with discrete actions. Their methodology uses CP to choose actions and does not involve evaluating target policies. Hence, the problem being considered is orthogonal to ours. There has also been concurrent work adapting CP to individual treatment effect (ITE) sensitivity analysis model [C 32). Similar to our approach, these works formulate the sensitivity analysis problem as one of CP under the joint distribution shift Px,y. While our methodologies are related, the application of CP explored in these works, i.e. ITE estimation under unobserved confounding, is fundamentally different.
Uncertainty in contextual bandits: Recall from the introduction, that most works in this area have focused on quantifying uncertainty in expected outcome (policy value) 5 Despite providing finite
on the expectation, these methods do not account for the variability in the outcome itself and in general are not adaptive w.r.t. X, i.c. they do not satisfy properties (i). (ii) from Sec. T 5 on the other hand, propose off-policy assessment algorithms for contextual bandits w.r.t. more general class of risk objectives such as Mean, CVaR etc. Their methodologies can be applied to our problem, to construct predictive intervals for off- off-policy outcomes. However, unlike COPP, these intervals are not adaptive w.r.t. X, i.e. do not satisfy property (i) in Sec. I.I Moreover, they do not provide upper bounds on coverage probability, which often leads to overly conservative intervals, as shown in our experiments. Lastly, while distributional perspective has been explored in reinforcement learning 3 no finite sample-guarantes are available to the best of our knowledge.
6 Experiments
Baselines for comparison. Given our problem setup, there are no established baselines. Instead, we compare our proposed method COPP to the following competing methods, which were constructed to capture the uncertainty in the outcome distribution and take into account the policy shift.
Weighted Importance Sampling (WIS) CDF estimator. Given observational dataset Dobs {zi, Oj yi Mohe proposed a non- parameeri WiS-based estimator for the empirical CDF of Y
under क. Fwis(t)= = "bbr pla, s. (H, where p(a, x) := (a]z) are the importance weights.
(ala)
Eist p(es.z))
We can use FWIs to get predictive intervals [ya/2, 91 /2/2 where ye Quantile, (aantile(Fwwrn (Fwis). The intervals [Ya/2, V1 V1-a/2] do not depend on æ.
7

---

Sampling Based Approach (SBA). As mentioned in Sec. 3.1 we can directly use the estimated P(y I, a) to construct the predictive intervals as follows. For a given weest we generate A, ltd. (.( gleat), and Y ~ P(- x A₂) for & L. We then define the predictive intervals for gteat using the a/2 and 1- a/2 quantiles of {Yi]ege. While SBA is not a standard baseline, it a natural comparison to make to answer the question of "why not construct the intervals using P(y |z, a) directly"?
6.1 Toy Experiment
We start with synthetic experiments and an ablation study, in order to dissect and understand our proposed methodology in more detail. We assume that our policies are stationary and there is overlap between the behaviour and target policy, both of which are standard assumptions 9 21
6.1.1 Synthetic data experiments setup
In order to understand how COPP works, we construct a simple experimental setup where we can control the amount of "policy shift" and know the ground truth. In this experiment, R. A E {1, 3, 4) and Y E IR, where X and Y ar, are normal random variables. Further details and additional experiments on continuous action spaces are given in Appendix D.I
Behaviour and Target Policies. We define a family of policies n. x)) where we use the parameter E (0, 1/3) to control the policy shift between target and behaviour policies. Exact form of (a a)) is given in D.1 For the behaviour policy x². use e² 0.3 (i.e xa(a x) = Wa.3 z)), and for target policies x², we use c" E {0.1, 0.2, 0.3]. Using the true behaviour policy, T², we generate observational data Dobs {z,, 0g y1 1=1r which is then split into training (Dtr) and calibration (Dont) datasets, of sizes m and respectively.
Estimation of ratios, (xx y). Using the training dataset Drr, we estimate P(ylæ, a) as P(y|z, a) N (u((z a), (zz a)), where u(a, a), o (2, a) are both neural networks (NNs). Similarly, we use NNs to estimate the behaviour policy n² from Dtr- Next, to estimate (xr y), use with h = 500.
Score. For the score function, we use the same formulation as in 18 i.e. s(x, v) - max (x)
y. Taas (z)), }. where (1r) denotes the / quantile estimate of P X trained using pinball loss.
Lastly, our weights w(x,, y) depend on x and y and hence we use grid of 100 equally spaced out y's in our experiments determine the predictive interval which satisfies (x) {y s(x, v) S Quantile;-o (F..))) This parallelizable and hence does not add much computational overhead.
Results. Table la shows the coverages of different methods the policy shift A, c² c increases. The behaviour policy n. = no.3 is fixed and we use n = 5000 calibration datapoints, across 10 runs. Table Ial shows, how COPP stays very close to the required coverage of 90% across all target policies compared to WIS and SBA. WIS intervals are overly conservative i.e. above the required coverage, while the SBA intervals suffer from under-coverage i.e. below the required coverage. These results supports our hypothesis from Sec. 3.1 which stated that COPP is less sensitive to estimation errors of P(ylar a) compared to directly using P(y| 22 a) for the intervals, i.e. SBA.
though WIS has reasonable coverage for A, = 0.0 and 0.1, the average interval length is huge compared to COPP. Fig. [ shows the predictive intervals for one such experiment with x= uo.1 and T² no.3. We can see that SBA intervals overly optimistic, while WIS intervals are too wide and are not adaptive w.r.t. X. COPP produces intervals which are much closer the oracle intervals.
Next, Table shows the mean interval lengths and
6.1.2 Ablation Study.
To isolate the effect of weight estimation error and policy shift, we conduct an ablation study, comparing COPP with estimated weights to COPP with Ground Truth (GT) weights and standard CP (assuming no policy shift). Table la shows that at A. = O, i.e. no policy shift, standard CP achieves the required coverage as expected. However the coverage of standard CP intervals decreases as the policy shift A, increases. COPP, on the other hand, attains the required coverage of 90%, by adapting the predictive intervals with increasing policy shift. Table [b shows that the average interval length of COPP increases with increasing policy shift A.. Furthermore, Table La illustrates that while COPP achieves the required coverage for different target policies, average it is slightly more
8

---

Table 2: Mean Coverage as function of policy shift A. and standard errors over 10 runs. COPP attains the required coverage of 90%, whereas the competing methods, WIS and SBA, are over- conservative i.e. coverage above 90%. addition, when we do not account for the policy shift, standard CP becomes progressively worse with increasing policy shift.
A, = 0.0
A. 0.1
A, 0.2
A, = 0.3
A. = =.4
COPP (Ours)
0.90 0.00
0.90 = 0.02
0.90 0.01
0.89
+ 0.01
0.1 = 0.01
WIS
1.00 + 0.00
1.00 0.00
0.92 + 0.00
0.94 0.00
0.91 + 0.00
SBA
0.00
0.00
0.97 0.00
CP (no policy shift)
0.91 + 0.02 0.92 0.02
0.93 0.01
0.94 + 0.01 0.96 + 0.01
conservative than using COPP with GT weights. This can be explained by the estimation error in w(( y). Additionally, to investigate the effect of integrating out the actions in 77. we also perform CP for each action a separately (as in [44 and then take the union of the intervals across these actions. In the union method, the probability of an action being chosen is not taken into account, (i.e..
intervals are independent of π and hence the coverage is overly conservative as expected.
Lastly, we investigate how increasing the number of calibration data affects the coverage for all the methodologies. We observe that coverage of COPP is closer to the required coverage of 90% compared the competing methodologies. Additionally, the coverage of COPP converges to the required coverage as n increases; see Appendix D.I.T for detailed experimental results.
6.2 Experiments on Microsoft Ranking Dataset
We now apply COPP onto a real dataset i.e. the Microsoft Ranking dataset 30k [16 space constraints, we have added additional extensive experiments on UCI datasets in Appendix D.3
25 4
Due to
Dataset. The dataset contains relevance scores for websites recommended to different users, and com- prises of 30, 000 user-website pairs. For each user-website pair, the data contains a 136-dimensional feature vector, which consists of user's attributes corresponding to the website, such as length of stay or number of clicks on the website. Furthermore, for each user-website pair, the dataset also contains a relevance score, i.e. how relevant the website was to the user.
First, given a user, we sample (with replacement) 5 websites from the data corresponding to that user. Next, we reformulate this into contextual bandit where a₂ E {1, 2, 3, 4, 5) corresponds to the action of recommending the a; 'th website to the user i. z; is obtained by combining the 5 user-website feature vectors corresponding to the user i.e. I2 E R5x136 4/1 (0, 1, 2, 4] corresponds to the relevance score for the a, 'th website, i.e. the recommended website. The goal is to construct prediction sets that are guaranteed contain the true relevance score with a probability of 90%.
Behaviour and Target Policies. We first train NN classifier model, fer mapping each 136- dimensional webr-we feature vector to the softmax scores for each relevance score class. We use this trained model fo to define family of policies which pick the most relevant website predicted by fe with probability and the rest uniformly with probability (1 - e)/4 (see Appendix D.2 for more details). Like the previous experiment, we use E to control the shift between behaviour and target policies. For n we use e 0.5 and for 'T" (0.1, 0.2, 0.3, 0.4, 0.5).
Estimation of ratios (XX Y). To estimate the P(y xx, a) we use the trained model fo as detailed in Appendix D.2 To estimate the behaviour policy tb we train neural network classifier model X + A. and we use to estimate the weights vxx y).
Score. The space of outcomes y in this experiment is discrete. We define prr (y x) =
Σ A - (())), A i). Using similar formulation as in 2 we define the score:
4 s(x, v)
Σ pm (s x))((" (y/ x) pm x))))) 3'
Since y' is discrete, we no longer need to construct a grid of values on which to compute Quantile, (f...). Instead, we will simply compute this quantity on each E V, when constructing the predictive sets (alest))
Results. Table 2 shows the coverages of different methodologies across varying target policies Te. The behaviour policy T No.5 is fixed and we use n 5000 calibration datapoints, across

---

10 runs. Table also shows that the coverage of WIS and SBA sets is dependent upon the policy shift, with both being overly conservative across the different target policies as compared to COPP. Recall that the WIS sets do not depend on glest and as result we get the same set for each test data point. This becomes even more problematic when Y discrete if, for each label y. P PXX,Y)PXX' (Y = v) > 10%, then WIS sets (with the required coverage of 90%) are likely to contain every label y E V. comparison, COPP able to stay much closer to the required coverage of 90% across all target policies. We have also added standard CP without policy shift as a sanity check, and observed that the sets get increasingly conservative as the policy shift increases.
Finally, we also plotted how the coverage changes as the number of calibration data n increases. We observe again that the coverage of COPP is closer to the required coverage of 90% compared the competing methodologies. Due to space constraints, we have added the plots in Appendix D.2
Class-balanced conformal prediction. Using the methodology described in Sec. 4.2
we construct
predictive sets, C% (I)), which offer label conditioned coverage guarantees (see B.4
i.e. for all
y E E..
F P(X,))-g; (Y C2 (X) -
We empirically demonstrate that Cy provides label conditional coverage, while Ön obtained using alg. may not. Due to space constraints, details on construction of C as well as experimental results have been included Appendix D.2.1
7 Conclusion and Limitations
In this paper, we propose COPP, algorithm for constructing predictive intervals on off-policy outcomes, which are adaptive w.r.t. covariates X. We theoretically prove that COPP can guarantee finite-sample coverage by adapting the framework of conformal prediction ments show that conventional methods cannot guarantee any user pre-specified coverage, whereas COPP can. For future work, it would be interesting to apply COPP to policy training. This could be step towards robust policy learning by optimising the worst case outcome
setup. Our experi-
We conclude by mentioning several limitations of COPP. Firstly, we do not guarantee conditional coverage in general. We outline conditions under which conditional coverage holds asymptotically (Prop. however, this relies on somewhat strong assumptions. Secondly, our current method estimates the weights w(x, y) through Py x, a). which can challenging. We address this limitation in Appendix B.5.2 where propose alternative method to estimate the weights directly, without having to model P(y x; a). Lastly, reliable estimation of our weights vi úi y) requires sufficient overlap between behaviour and target policies. The results from COPP may suffer in cases where this assumption violated, which we illustrate empirically in Appendix D.1.2 We believe these limitations suggest interesting research questions that we leave to future work.
Acknowledgements
We would like to thanks Andrew Jesson, Sahra Ghalebikesabi, Robert Hu, Siu Lun Chau and Tim Rudner for useful feedhack. JFT is supported by the EPSRC and MRC through the Ox WaSP CDT programme (EP/L001670//)) MFT acknowledges his Phl) funding from Google DeepMind. RC and AD are supported by the Engineering and Physical Sciences Research Council (EPSRC) through the Bayes4Health programme [Grant number EP/R018561/I].
References
[1] Jason Altschuler, Victor-Emmanuel Brunel, and Alan Malek. Best arm identification for contaminated bandits. Journal of Machine Learning Research, 20(91):1-39, 2019.
[2] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[3] Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International Conference Machine Learning, pages 449-458 2017.
10

---

[4] Alberto Bietti, Alekh Agarwal, and John Langford. A contextual bandit bake-off. arXiv preprint arXiv:1802.04064, 2018.
[5] Yash Chandak, Scott Niekum, Bruno Castro da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off off-policy evaluation. arXiv preprint arXiv:2104. /2820, 2021.
[6] Miroslav Dudik, Dumitru Erhan, John Langford. and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4), 2014.
[7] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447-1456.
PMLR, 2018.
[8] Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution- conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-482, 2021.
[9] Audrey Huang, Liu Leqi, Zachary C. Lipton, and Kamyar Azizzadenesheli. Off-pooicy risk assessment in contextual bandits. arXiv preprint arXivv2104.08977 2021.
[10] Ying Jin, Zhimei Ren, and Emmanuel Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. arXiv preprint arXiv:2111.122 2021.
[11] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In AAAI Conference Artificial Intelligence, volume 34, pages 4436-4443, 2020.
[12] Ilja Kuzborskij, Claire Vernade, András György, and Csaba Szepesvári. Confident off-policy evaluation and selection through self-normalized importance weighting. In International Conference Artificial Intelligence and Statistics, pages 440-648, 2021.
[13] Jing Lei and Larry Wasserman, Distribution-fiee free prediction bands for non non-parametric regression. Journal of the Royal Statistical Society: Series B, pages 71-96, 2014.
[14] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B, pages 911-938, 2021.
[15] Muhammad Osama, Dave Zachariah, and Peter Stoica. Learning robust decision policies from observational data. arXiv preprint vrXiv:2006.02355, 2020.
[16] Tao Qin and Tie-Yan Liu. Introducing LETOR 4.0 datasets. arXiv preprint arXiv: 2013.
[17] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel J. Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2(2), 4 2020.
[18] Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, pages 3543-3553, 2019.
[19] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
[20] David Stutz, Krishnamurthy Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. International Conference on Representation Learning, 2022.
[21] Yi Su. Maria Dimakopoulou. Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. CoRR, abs/1907. Abs/1997.09623,, 2019.
[22] Yi Su, Maria Dinakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Muchine Learning, pages 9167-9176. PMLR, 2020.

---

[23] Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16(52):1731- 1755, 2015.
[24] Adith Swaminathan and Thorsten Joachims. The normanormllized estimator for counterfactual learning. In Advances in Neural Information Processing Systems, volume 28, 2015.
[25] Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miroslav Dudík, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems, 2017.
[26] Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off oiflcollcy evaluation. In AAAI Conference Artificial Intelligence, 2015.
[27] Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, 2019.
[28] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Steven C. H. Hoi and Wray Buntine, editors, Proceedings of the Asian Conference on Machine Learning, volume 25 of Proceedings of Machine Learning Research, pages 475-490, Singapore Management University, Singapore, 04-06 Nov 2012. PMLR.
[29] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning Learning a Random World. Springer Science & Business Media, 2005.
[30] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudík. Optimal and adaptive off-policy evalua- tion in contextual bandits. In International Conference on Machine Learning, page 3589-3597, 2017.
[31] Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alhéé-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[32] Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. arXiv preprint /tti::2//2.003999 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all thcoretical results? [Yes] See section 4
(b) Did you include complete proofs of all theoretical results? [Yes] See section A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as URL)? [Yes] See section
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See sections D
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See sections
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section D
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zOYIps_b4X", "reviewer": "Reviewer_cY8Q", "summary": "This paper studies conformal prediction for the (mixed) outcome in contextual bandits under a target policy, using data from a different behavior policy. To tackle the shift from observations to targets, the authors provide the explicit formula for the distributional shift, then propose to use density estimation as well as a Monte Carlo sampling method to compute the conformal prediction intervals. The authors also build on existing methods to show the marginal coverage and asymptotic conditional coverage of the proposed methods. Finally, extensive experiments show the performance of the proposed method. ", "questions": "1. How does the miscoverage depend on the estimation error of $\\hat{P}(y|x,a)$ instead of the hard-to-interpret $\\hat{w}$? \n\n2. The density estimation of $\\hat{P}(y | x,a)$ is known to be hard because of data scarcity (more difficult than propensity score estimation, at least). It would be helpful to justify this approach by referring to some classical literature on estimation performance under appropriate conditions. \n\n3. Please provide more explanation for comparing to SBA in your experiments. ", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \n\nThis paper studies a new problem. The techniques contain some novel points but somehow heavily rely on established results. The approach and the results are not too surprising when reading the paper. \n\n\n----- \nSignificance: \n\nThis paper studies a (moderately new) problem of quantifying the uncertainty in outcomes from contextual bandits. Instead of predicting counterfactuals, the authors consider the mixed overall outcome. This adds to the application of conformal inference, but I am not sure whether this is indeed interesting. \n\n\n-----\nQuality: \n\nThe overall quality of this paper is good. The presentation is clear; the theoretical results are easy to understand with nice interpretation and discussions; the comparison to the literature is detailed and comprehensive; the conclusions from experiments are clear. \n\nBesides the strengths, I do have a few concerns. \n\n1. Theories involving weight function estimation. \n\nIt is unclear to me how it depends on the Monte Carlo approximation with $h$ samples. Although Proposition 4.2 provides an error bound that is quite similar to Lei and Candes (2021), I am a bit concerned that it is not the most appropriate for this place. Instead, it might be more informative if the error bound could be in terms of $\\hat{P}(y | x, a)$ because this quantity is estimated, while $\\hat{w}$ is built on top of it. \n\n2. Estimation of $\\hat{P}(y | x,a)$. \n\nThe density estimation of $\\hat{P}(y | x,a)$ is known to be hard because of data scarcity (more difficult than propensity score estimation, at least). It would be helpful to justify this approach by referring to some classical literature on estimation performance under appropriate conditions. \n\n\n3. Comparison to SBA.\n\nI am not convinced why the authors compare their method to SBA. It is not a conformal-based algorithm and not studied in the literature. \n\n-----\nClarity: \n\nThe presentation of this paper is clear and easy to follow. The results are interpreted clearly from time to time. One concern is for the weight function estimation, as stated above. \n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "vKt9kO8tmY", "reviewer": "Reviewer_rDgw", "summary": "This paper studies off-policy value predictions at particular X with either marginal or conditional coverage guarantees.  Unlike prior art, this paper preserves the context distribution and models the shift in the outcome distribution via the shift in the action distribution.   The methodology thus applies to stochastic policies and in infinite action spaces (with some caveats).   For marginal guarantees, the methodology works with either a known historical policy or estimated propensity scores.  As is typical in conformal prediction, conditional coverage guarantees require restrictive additional assumptions.", "questions": "This paper repeatedly states that known weights are not typical, which contradicts my extensive personal experience with large scale recommendation systems with known randomization.  In causal modeling with observational data propensities are often estimated, but the paper emphasizes the contextual bandit application area.   The estimated weight results are appreciated; but for instance the paper misses a chance to present a simplified corollary of proposition 4.3 for the case when w(x,y) is known.\n\nThe moment condition in proposition 4.2 is presented without comment.  Some discussion about how this affects the estimation of \\hat{w} would be appreciated (e.g., suggest a regularizer; or comment upon the moment properties of equation (7)).", "limitations": "This reviewer is satisfied.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is in the happy position of (essentially) being the first proposed solution to a problem of clear practical importance.\n\nThis suffers from the page limit.  Personally, I would elide more of the requisite but uninteresting experimental details to the appendix in order to expand the discussion around proposition 4.3 in the case of known weights.  Alternatively, section 2.2 is ultimately unnecessary: yes prior art used a different covariate shift model, but you could reduce this to a sentence or two in the related work and perhaps a reference to the appendix, and get back some valuable space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Cqwfv1tLMe2", "reviewer": "Reviewer_E2XW", "summary": "The paper studies how to construct predictive intervals from a contextual bandit policy such that the true labels are within the predictive intervals with a user specific probability, using some bandit feedback data, which they call the calibration data. \n\nThe paper identifies that the data distributions are different under the logging policy and the target policy in contextual bandits, which impedes using conventional conformal prediction methods to provide predictive intervals with coverage guarantees. So they adopt the conformal prediction method with covariate shifts [13] to this problem to provide coverage guarantees despite the data distribution shift. \n\nThey provide marginal coverage (adapted from prior works) and conditional coverage analysis (with strong assumptions) of the method, and demonstrate its effectiveness in empirical evaluation. ", "questions": "See weaknesses. ", "limitations": "I did not find limitations and potential negative societal impact. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. \n\nThey provide coverage analysis of the proposed method. \n\nThe empirical evaluation demonstrates the effectiveness of the proposed method. \n\nWeaknesses:\n\n1. The method does not provide any optimality guarantee. The method does provide coverage guarantee (i.e. the prediction set contains the true label with high probability). But it does not have any guarantee on the size of the prediction sets. I can easily construct prediction sets with perfect coverage guarantee, i.e. the set of all labels. \n\n2. The paper does not seem to provide new technical contributions. The proofs seem to be directly adapted from prior works. \n\n3. I am also a bit confused about whether the guarantees are desirable. In proposition 4.1, the randomness is in both the calibration data and the one test data point, right? Does this mean that for each single test data, we need n calibration data to construct the prediction sets to provide the coverage guarantee? Would it be possible to get this kind of guarantee instead: with high probability in the calibration data, the probability that the true label is in the prediction set is greater than a user-specified level 1-\\alpha? If this is true, it then makes sense to construct a single set prediction function using calibration data, and then apply it to a set of test data. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept