Post-Contextual-Bandit Inference
Authors: Aurelien Bibaut
Maria Dimakopoulou
Nathan Kallus
Antoine Chambaz
Mark van der Laan
Venue: NeurIPS 2021 Poster (2021)
Keywords: contextual bandit
causal inference
confidence intervals
adaptive data collection

Abstract: Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking because they can both improve outcomes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end of the study, nonetheless, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or value of new policies. The adaptive nature of the data collected by contextual bandit algorithms, however, makes this difficult: standard estimators are no longer asymptotically normally distributed and classic confidence intervals fail to provide correct coverage. While this has been addressed in non-contextual settings by using stabilized estimators, variance stabilized estimators in the contextual setting pose unique challenges that we tackle for the first time in this paper. We propose the Contextual Adaptive Doubly Robust (CADR) estimator, a novel estimator for policy value that is asymptotically normal under contextual adaptive data collection. The main technical challenge in constructing CADR is designing adaptive and consistent conditional standard deviation estimators for stabilization. Extensive numerical experiments using 57 OpenML datasets demonstrate that confidence intervals based on CADR uniquely provide correct coverage.

Post-Contextual-Baniii Inference
Aurélien Bibaut
Maria Dimakopoulon Netflix
Nathan Kallus
Netflix abibaut@netflix.comm com
Cornell University and Neuflix kallus@cornell. edu
mdinakoppullou@hetllixccoom
Antoine Chambaz
Mark van der Laan
Université de Paris
University of California, Berkeley
antoine.. chambaz@u-paris.
laan@stat berkeley edu
Abstract
Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking because they can both improve out- comes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end the study, nonetheless, we still want to construct valid confidence intervals av- erage treatment effects, subgroup effects, or value of new policies. The adaptive nature of the data collected by contextual bandit algorithmss, however, makes this difficult: standard estimators are no longer asymptotically normally distributed and classic confidence intervals fail to provide correct coverage. While this has been addressed in non-contextual settings by using stabilized estimators, variance- stabilized estimators in the contextual setting pose unique challenges that we tackle for the first time in this paper. We propose the Contextual Adaptive Doubly Robust (CADR) estimator, novel estimator for policy value that is asymptotically normal under contextual adaptive data collection. The main technical challenge in constructing CADR designing adaptive and consistent conditional standard deviation estimators for stabilization. Extensive numerical experiments using 57 OpenML datasets demonstrate that confidence intervals based CADR uniquely provide correct coverage.
Introduction
Contextual bandits, where personalized decisions are made sequentially and simultancously with data collection, are increasingly used to address important decision-making problems where data limited and/or expensive collect, with applications in product recommendation [Li et al.) 2010), revenue management [allus and Udell] 2020 Qiang and Bayati] 2016), and personalized medicine (Tewari and Murphy 2017]. Adaptive experiments, whether based on bandit algorithms Bayesian optimization, are increasingly being considered in place of classic randomized trials in order to improve both the outcomes for study participants and the chance of identifying the best treatment allocations (Athey et al. 2018 Quinn et al. 2019 Kasy and Sautmann, 2021 Bakshy al. 2018).
But, at the end of the study, we still want to construct valid confidence intervals on average treatment effects. subgroup effects, or the value of new personalized interventions. Such confidence intervals are, for example, crucial for enabling credible inference the presence or absence of improve- ment of novel policies. However, due to the adaptive nature of the data collection, unlike classic randomized trials, standard estimates and their confidence intervals actually fail to provide correct coverage, that is, contain the true parameter with the desired confidence probability (e.g., 95%). A variety of recent work has recognized this and offered remedies (Hadad et al., 2019, Luedtke and van der Laan 2016), but only for the case non-contextual adaptive data collection. Like classic
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

confidence intervals, when data comes from a contextual bandit or any other context-dependen adaptive data collection these intervals also fail to provide correct coverage. In this paper, we propose novel asymptotically normal estimator for the value of a (possibly contextual) policy from context-dependent adaptively collected data. This asymptotic normality leads directly to the construction of valid confidence intervals.
Our estimator takes the form of a stabilized doubly robust estimator, that is, a weighted time average of an estimate of the so-called canonical gradient using plug in estimators for the outcome model, where each time point is inversely weighted by its estimated conditional standard deviation given the past. We term this the Contextual Adaptive Doubly Robust (CADR) estimator. We show that, given consistent conditional variance estimates which at each time point only depend on previous data, the CADR estimator is asymptotically normal, and as result we can easily construct asymp- totically valid confidence intervals. This normality is in fact robust to misspecifying the outcome model. A significant technical challenge is actually constructing such variance estimators. We re- solve this using an adaptive variance estimator based on the importance-samplnng ratio of current to past (adaptive) policies at each time point. We also show that we can reliably estimate outcome models from the adaptively-collected data so that we can plug them in. Extensive experiments us- ing 57 OpenML datasets demonstrate the failure of previous approaches and the success of ours at constructing confidence intervals with correct coverage.
1.1 Problem Statement and Notation
The data. Our data consists of a sequence of observations indexed 1..... . T comprising of context (() X, action A(t) E A. and outcome Y() E y C R generated by an adaptive experiment, such as a contextual bandit algorithm. Roughly, at each round / = 1,2,... ..,,,..,7 T, an agent formed contextual policy 9:(a a:) based on all past observations, then observed an independently drawn context vector (t) ~ Qo.x. carried out an action A(t) drawn from its current policy ge (()),, and observed outcome (t) ~ Qo A(t), X (t)) depending only on the present context and action. The action and context measurable spaces X, A are arbitrary, e.g., finite or continuous. We assume outcomes are bounded and, without loss of generality, take y [-1,
More formally, we let O(t) == (x),) A(t), Y()) and make the following assumptions about the sequence O(1).... OTT) comprising our dataset. First, we assume ((t is independent of all else given A(L) and has a time-independent marginal distribution that we denote by Qo Second, we assume A(l) is independent of all else given O(1), O(t - 1), (() and we set ge(- X(()) to its (random) conditional distribution given O(1), O( 1), (1). Third, we assume (() is independent of all else given X (t), A(t) and has time- imee-nndependent conditional distribution given ((t = z, A(t) = a that is denoted by (- X). The distributions Qo.x and Qo.y are unknown, while the policies (a z) are known, as would be the case when running an adaptive experiment. To simplify presentation we endow A with a base measure μl (e.g., counting for finite actions or Lebesgue for continuous actions) and identify policies ge with conditional densities with respect to (w.r..) P.A In the case of K < co actions, policies are maps from X' to the K K-simplex.
Note that, as the agent updates its policy based on already collected observations, Jt is a random O(1).... O(L 1 meeauraalee object. This is the major departure from the setting considered in other literature on off-policy evaluation, which only consider a fixed logging policy, Ir = g. that is independent of the data. See Section 1.2
The target parameter. We are interested in inference on generalized average causal effect ex- pressed as functional of the unknown distributions above, Vo = (Qu,x, Q,,y), where for any distributions Qx, Qy we define
([ Qr)= - yQx (dx)g))) (a x)dp (dy a,
where ga 2) : A + G- is a given fixed, bounded function. Two examples are: (a) when g" is policy (conditional density), then vo is its value; (b) when g" is the difference between two policies then Vo the difference between their values. prominent example of the latter is when A } and g (a x) a, which is known as the average treatment effect. If we include an indicator for x being some set, then we get the subgroup effect. Defining the conditional mean outcome,

---

Qo(a,z) BQo (1-x rdy a,x),
we note that the target parameter only depends on Q0,y via Qu. so we also overload notation and write YP (Qx. , Q) = Q(a, x)Qx (dx) (dx)g(a x)) for any function Q A X + y. Note that when |A] < and Ma is the counting measure, the integral over a is simple sum.
Canonical gradient. We will make repeated use of the following function: for any conditional density (a. .) g(a æ)) any probability distribution Qx over the context space X, and any function Q A x X, we define the function D' Q) + by
ga(a x) σ)
(y ()a, + Q(d', x))" =))dua(a').
g(a
Further, define Dig, canonical gradient of the target parameter V w.r.t the usual nonparametric statistical model com- prising all joint distributions over [van der Vaart 2000, van der Laan and Robins, 2003).
Qx D', Qx Q) VQQx. Q), which coincides with the so-called
Integration operator notation. For any policy g and distributions Qx Qy denote by Pq.g the IR, we use the integration
induced distribution on O == x x y. For any function operator notation
oo.f = / (xx 2)dpa(a))Y(dt a,
that is, the expectation w.r.t. PQ.g alone. Then, for example, for any O(.)..... O(s 0(s-1) --asuurabll random function f R. we have that PQo.g. Equ.g. [f(((s)) O(1). 1)].
1.2 Related Literature and Challenges for Post-Contextual-Banditt Inference
Off-policy evaluation. In non-adaptive settings, where 9: = is fixed and does not depend on
previous observations, common off-the shelf estimators for the mean outcome under g include the Inverse Propensity Scoring (IPS) estimator [Beygelzimer and Langford 2009 Li et al. 2011 and
and the Doubly Robust (DR) estimator [Dudík al. 2011 Robins et 1994]
IPPS
1 T D'..O).
DR
1 T
Σ ('
where 2 is estimator of the outcome model Qo(a. If we cross-fitting to estimate Q CChrnozzuukv et al.] 2018). then both the IPS and DR estimators are unbiased and asymptotically normal, permitting straightforward inference using Wald confidence intervals (i.e., 1.96 of the estimated standard error). There also exist many variants of the IPS and DR estimators that, rather
than plugging the importance sampling (IS) ratios (g /9:)(((()) X (t)) and/or outcome-model estimators, instead choose them directly with the aim to minimize error [e.g. Kallus 2018 Farajtabar
al. 2018 Thomas and Brunskill 2016 Wang et 2017 Kallus and Ueharal 2019
Inference challenges in adaptive settings. In the adaptive setting, is easy to see that, if in the tth term for DR we use an outcome model Q using only the observations O(1)..., 0(t O(t 1), then both the IPS and DR estimators both remain unbiased. However, neither generally converges normal distribution. One key difference between the non-adaptive and adaptive settings that the IS ratios (g*/gs (0//9:((A)))) (A(L) (L)) can both diverge to infinity converge zero. As result of this, the above two estimators may either be dominated by their first terms or their last terms.
more theoretical level, this violates the classical condition of martingale central limit theorems that the conditional variance of the terms given previous observations stabilizes asymptotically.
Stabilized estimators. The issue for inference due to instability of the DR estimator terms was recognized by [Luedtke and van der Laan (0166) in another setting. They work in the non-adaptive setting but consider the problem of inferring the maximum mean outcome over all policies when the optimal policy is non-unique. Their proposal is a so-called stabilized estimator, in which each term is inversely weighted by an estimate of its conditional standard deviation given the previous terms.

---

This stabilization trick has been also been reused for off- inference from non-contextual bandit data by Hadad al.] [019], as the stabilized estimator remains asymptotically normal, permitting inference. In their non-contextual setting, an estimate of the conditional standard deviation of the terms can easily be obtained by the inverse square root propensities. In contrast, in our contextual setting, obtaining valid stabilization weights is more challenging and requires a construction involv- ing adaptive training on past data. Zhang et al. [2021] use similar weights to Hadad et al. [2019] in order to stabilize an -essimattor fit to adaptively collected data for the purpose of inference on a true model for Qo- While they do consider a contextual setting, their method requires correct specification, else the inferential target depends on the logging policies. We focus only on off-policy
evaluation but do not require well-specified models.
Concurrent work of Zhan et al. [2021]. The paper of Zhan et al. 2021). which was published after the submission of our paper, is also motivated by establishing an unbiased and asymptoti- cally normal estimator from data collected by contextual bandit. Both method and theirs rely on adaptively weighted DR scores and yield, under certain conditions, asymptotically normal es- timators. However, the way they construct their adaptive weights is quite different from ours. In particular, their adaptive weights at time I (a) do not incorporate an outcome model and do not rely on it and (b) depend only on the known propensity score, which is plugged-in. As result of these different constructions, our analysis and theirs require different sets of conditions. While we require slightly more stringent conditions on the exploration rate and the complexity of the logging policy class than they do (our assumptions 5 and 6). they require some form of asymptotic stabilization of the logging policies (their conditions (8) and (12)). An instance where the logging policy does not stabilize is, for example, when various actions have the same expected reward on certain subsets weights for the
of the context space. Our approach in this paper is to use the suriance-stabilizaii contextual bandit setting, just like Hadad et al. [2019] do the in the non-contextual bandit setting and Luedtke and van der Laan] [2016] do an nid setting with multiple optimal policies. Although Zhan et (2021] also tackle the contextual bandit setting, they do not use the variance-stabillization weights; correspondingly their conditions and results apply in different, complementary settings. The key challenge we tackle in applying auriance-stabilizatio weights in the contextual bandit set- ting is dealing with the need to use sequentially estimated weights.
1.3 Contributions
In this paper, we construct and analyze a stabilized estimator for policy evaluation from context- dependent adaptively collected data, such as the result of running a contextual bandit algorithm. This then immediately enables inference. After constructing generic extension of the stabilization trick, the main technical challenge is to construct a sequence of estimators o....., Ôg of the conditional standard deviations that are both consistent and such that for each t, ã; only uses the previous data points O(1),... ..... O - 1). We show in extensive experiments across a large set of contextual bandit environments that confidence intervals uniquely achieve close to nominal coverage.
2 Construction and Analysis of the Generic Contextual Stabilized Estimator
In this section, we give a generic construction of a stabilized estimator in our contextual and adaptive setting. That is, given generic plug-ins for outcome model and conditional standard deviation. We then provide conditions under which the estimator is asymptotically normal, as desired. To develop CADR, we will then proceed to construct appropriate plug in estimators in the proceeding sections.
2.1 Construction of the Estimator
Outcome and variance estimators. Our estimator uses a sequence (2.)21 of estimators of the
outcome model Qo. such that, for every 6, Q, is O(1),... .., ((t)-measurable that is, is trained using only the data up to time i. A key part of our estimator is the conditional variance estimators. Also, require estimates of the conditional standard deviation of the canonical gradient. Define:
Fo. == (9t),
where [ge(s)) == Varqo.a (O))) 100)...,.
4

---

Let (oz) be given sequence of estimates of o.r such that o; is OI,).... that is, is estimated using only the data up to time
O(t ((((t-)--mmaumane measurabe,
The generic form of the estimator.
The generic contextual stabilized estimator is then defined as:
T
of Do,
T
2.2 Asymptotic normality guarantees
We next characterize the asymptotic distribution of u under some assumptions. Assumption (Non degenerate efficiency bound). inf, P (g, Qu, Qa,x) > 0.
D states that there no fixed logging policy g such that the efficiency bound for es- timation of V Qu Qx.x) in the nonparametric model, from i..d. draws of Poo.g- is zero. If as- sumption does not hold, there exists a logging policy 9 such that, if O = (X, A) ~ PQu.9> then (g" (A | X))Y equals (QQ,, Qo,x) with probability 1. In other words, as- sumption does not hold, there exists logging policy g such that (QQ. Qo.x) can be esti- mated with no error with probability 1 from a single draw of Pqo.g. Thus, it is very lax. An easy sufficient condition for Assumption Tis that the outcome model has nontrivial variance in that Varqa, x (J (a, X)g"((( (a (a)) > 0.
Assumption
Assumption (Consistent standard deviation estimators.). 6, Jo,t 1000, 0 almost surely.
In the next section we will proceed to construct specific estimators o, that satisfy Assumption 2 leading to our proposed CADR estimator and confidence intervals.
Assumption (Exploration rate).
suuuaeA,,xx 9 (a =)]l w(-)
Assumption 3 requires that the exploration rate of the adaptive experiment does not decay too quickly. Crucially, Assumption requires that have nontrivial propensities of pulling any one arm in any one context, conditioned on the history. While this may be satisfied by inherently ran- domized algorithms such e-greedy and Thompson sampling, this is not satisfied by algorithms such as UCB, which are deterministic once given the history. This limitation is the same as for Hadad et al. [2019] Zhan al. 2021].
Remark 1. One can weaken Assumption assumption to just requiring w((-1) decay by instead slightly strengthening Assumption Too require that ob, Z || A Dr (a | loo.. which would be implied by g" placing some mass on actions with decaying propensity and Y having everyvhere-nontrivval conditional variance. Here 2 by means that for some constant c. >> O, we have a, cbç for all t 2 1. However, as we will in fact strengthen Assumption 3 anyway in Assumption O below, which will use in constructing estimators oe that satisfy Assumption 2 do not discuss this potential weakeming of Assumption 3
Based on these assumptions, we have the following asymptotic normality result:
Theorem 1. Denote IT = (T-1 DT 1 Under Assumptions 7 to it holds that
T T (ūr-vo) -v) vo) $ 11)
Remark 2. Theorem does not require the outcome model estimator to converge at all. As we will see in Section 3 our conditional variance estimator does require that the outcome model converges a fixed limit Q₁ but this limit does not have to be the true outcome model Qo- In other words, consistency of the outcome model is not required at any point of our analysis.
3 Construction of the Conditional Variance Estimator and CADR
We now tackle the construction of o; satisfying our assumptions; namely, they must be adaptively trained only past data at each t and they must be consistent. Observe that ob, =g( Q,-1),
5

---

Algorithm The CADR Estimator and Confidence Interval
Input: Data O(I).... OTT), policies 91....19T larget g", outcome regression estimator
for t = 1,2...., '''do
Train Q,-1 on O11,,.............,,... ................................ O(t - 1) using the outcome regression estimator
Set Di.5 (g Q ((())((t) for t,..., // (note index order compared next line)
Set o
.- Σ (A({)))))()) (x(d(t)((())
-2 - E (A(e)) Di,,)
N. (. X(3)
end for
-
Set
I'y
+ Σ 0,,
Return estimate
1 bf Σl= o2 Dí,, and confidence intervals Cl = Si
where we define
oglg, Q) = (0.2 Q) - ( Q))², 2o Q)= = (D") Q), i i= 1,2.
Designing an ((1). O(t 1) measuraaee estimator of ob.: presents several challenges. First, while we can only use observations O(1),... Ott 1) to estimate it, ob, is defined as a function of integrals w.r.t. Poo... from which we have only one observation, namely O(t). Second, our estimation target ob.- = og(e Qi-1) random as it depends on Jr and Qj Third, 9t, Qe depend on the same observations O(1), Ott 1) that we have at our disposal to estimate o².-
Representation via importance sampling. We can overcome the first difficulty via importance sampling, which allows us to write lo Q), i = 1,2 as integrals w.r.t. Poo.g, 8 = 1,..., 1, i.e,, the conditional distributions of observations O(s), 8 I,...,l L -
given their respective past.
Namely, for any s 2 i 1,2, we have that
Lo Q) Poo..
9 (1') Q).
ga
Dealing with the randomness of the estimation target. We turn second challenge. Since ob, can be written in terms of ( 1i-1) for i = 1,2, Eq. suggests perhaps an approach based on sample averages of (9)/9,) 2 over s. However, whenever < t, the lat- Oft memauuurae function due to the dependence on 9r and Q. Namely, Q.-1)} does not coincide in general with the conditional expectation Equ.a. [(19 (D') (O(s)) O(s 1)], as would arise from a sample average. We now look at solutions to overcome this difficulty, considering first Q.-1 and then gr.
ter is an Po..g.
Dealing with the randomness of Q, We propose estimator of of 2(-1) for any fixed g- While requiring that Q,- converges to the true outcome regression function Qu is strong require- ment, most reasonable estimators will at least converge to some fixed limit Q1. As result, under an appropriate stochastic convergence condition on (Q,-1)221, &o,(gg Q,-1) can be reasonably approximated by the corresponding Cesaro averages, defined for i 1,2 as
(o... (0) 1
La,(g - Σ Equ.s.
Ö(s 1)]
These are easy to estimate from the corresponding sample averages, defined for i = 1,2 as
(lg/ - (
( - 6=:())==111
since for each i = 1, 2, the difference (o) - 0,. (g) is the average of a martingale difference sequence (MDS). We then define our estimator of og(g. Q,,1) as
@x(g) = t2.(9) ?
(3)

---

From fixed to random It- So far, we have proposed and justified the construction of (g) as an estimator of (g. Q2-1) for a fixed g. We now discuss conditions under which ã (g() is valid estimator of (9t Qc-1). When g is fixed, for each i = 1,2, the error (g) Po Q,-1) decomposes the sum of the MDS average (g) - (g) and of the Cesaro approximation error (g) (g. 11-1). Both differences are straightforward to bound. For a random St. the term (g:) (ge)) is no longer an MDS average. Fortunately, under complexity condition on the logging policy class 9. we can bound the supremum of the martingale empirical processes { (g) (g)) 9 E G}. which in turn gives us a bound on ( L
Consistency guarantee for की. Our formal consistency result relies on the following assumptions. Assumption 4 (Outcome regression estimator convergence). There exists 3 > 0, and a fixed func- tion Q1 A X + R such that 1|Ê, Q1 21..1.0o.x.9" P(- almost surely.
The next assumption is a bound on the bracketing entropy (see, e.g., van der Vaart and Wellner 1996) for definition) of the logging policy class.
Assumption 5 (Complexity of the logging policy class). There exists a fixed reference function ga x), a class of conditional densities G, and some p > O such that gr. E &G 2 1 almost surely, supgeg ll9// 1100 S G, /gg 1lc S G, and
log log, -l (l2.0o...gat) SED
where G/gnt == {g/g"0t ge
Next, we require condition on the exploration rate that stronger than Assumption 3
Assumption 6 (Exploration rate (stronger)). ||supaeA. ged x)/9((a I 2) S (8()))
where (333 p) == min(1/(3 p), + 3).
Theorem 2. Suppose that Assumptions 4 4o hold. Then, oī - o(1) almost surely.
Remark 3. While we theoretically require the existence of a logging policy class 9 with controlled complexity, we do not actually need to know G to construct our estimator. Moreover, while we require bound on the bracketing entropy of the logging policy class G. we impose no restriction on the outcome regression model complexity, permitting us to use flexible black-box regression methods. Usually we simply take gf (a x) so the boundedness in Assumption 5 is simply boundedness of densities, but the above allows for slightly more flexibility in cases where we do not necessarily explore all actions equally well in Assumption
Remark 4. Assumption 4 requires (Qt) be sequence of regression estimator, such that for every
t2 1. Q, is fitted on O(L), Ott) and for which we can guarantee a rate of convergence to some fixed limit Q1- Note that this can at first glance pose a challenge since observations O(1),..., O(t) are adaptively collected. In the appendix, we give guarantees for outcome regression estimation over a nonparametric model using an importance sampling weighted empirical risk minimization.
Remark 5. Assumptions Daniionn restrictions on the logging policy, that is, the contextual bandit algorithm that was run. Assumption is not very restrictive, especially for large It is generally satisfied for p arbitrarily close for e-greedy with outcome regressions in a VC-subgraph class such as trees or for Thompson sampling with parametric models. For larger it can accommodate very general policy classes, even Donskee classes (p > 2). Assumption b on the other hand, may be restrictive as it can require a lot of exploration. While it appears that at least w(1/t) explo- ration might be necessary for some normality as Theorem I (see Remark [7. the rates needed in A Assumption 6 may be too strong than actually needed. We therefore view it as high-level assump- tion, as it does not change the method itself and may not need to be satisfied in practice, and we leave it as an open question how to tighten our analysis to be able to rely on less exploration.
CADR asymptotics. Our proposed CADR estimator is now given by plugging our estimates O from Eq. 33) into Eq. ([. as summarized in Algorithm [ As an immediate corollary of Theorems [ 2 we have our main guarantee for this final estimator, showing CADR is asymptotically normal, whence we immediately obtain asymptotically valid confidence intervals.
7

---

Corollary (CADR Asymptotics and Inference). Suppose that Assumptions and to hold. Let oe be given as in Eq. 3 Denote TT = (T- 1--Ei=1 Ei=1 o. Then,
TT V (T por-Vo (wr-Vo) vo) 1 (0,,
Moreover, letting Sa denote the a-quantile of the standard normal distribution,
Pr (QQu.x Qu) E ( + -1-0///// T- 1
4 Empirical Evaluation
In this section and in sections E.2 E.5
the appendix, we present an extensive and reproducible empirical evaluation on public datasets that demonstrates the robustness of CADR confidence inter- vals using contextual bandit data with comparison to several baselines.
To produce the 8 figures in the paper (Figure Tin in the main paper and Figures 2 in the appendix) we worked with: 57-72 real-world classification datasets (OpenMI. Curated Classification beneh- marking suite 2018 [Bischl et al.] 2017]) converted to contextual bandit datasets (wide-spread eval- uation approach in the offline and online contextual bandit literature); 4 different target policies; estimators (CADR, and 5 other popular baseline estimators in the literature); 8 different training procedures (sequential cross-fitting vs. cross-time cross-fitting in Figures [ndd] and 3 misspecified vs. well-specified outcome model family in Figures + and weighted vs. unweighted outcome model fitting in Figures 6 and Z large data vs. small data in Figures 3 and R). For each (dataset, target pol- icy, estimator, training peoceddure--oombinaee standard errors of the estimators' performance.
64 simulations were run to obtain and report tight
In the subsections that follow, we present our approach and the main results.
4.1 Baseline Estimators
We compare CADR to several benchmarks (our experiments focus the case of finitely-many actions, A = {L,...,K] K.., }). All baselines take the following form for a choice of We, we, Q;
1 wt
1 T
ΣT w/ w 1)2
V
w. D',
Cl.
V + 61-0/2
T
E WA
1=1
where D D/=wal w) (() + Σ
Ô (a, ((t))9' X(t)).
6=1
The Direct Method (DM) sets wy = 1, = and fits Q,, by running some regression method for each the data ((X(s) : S S 1, a). We will use either linear regression or decision-tree regression, both using default sklearn parameters. Note that
even in non-contextual settings, where QT1 is simple per-arm sample average, Q may be biased due to adaptive data collection [Xu et al. 2013 Luedtke and van der Laan 2016 Bowden and Trippa 2017 Nie et al.) 2018 Hadad al. 2019 Shin al. 2019). Inverse Propensity Score Weighting (IPW) sets we = 1, = (g" (A(() X()),, Q = O. Doubly Robust (DR) sets
W/ = 1, w (g" 9t (A(t) X(t)) and fits Q.- in DM. More Robust Doubly Robust (MRDR)
2018 is the same as DR but when fitting Q.11 we reweight each data point Farajtabar et al.
by 9 [1
X(x)))) None of the above are generally asymptotically normal under 9. [A(+)]
adaptive data collection (adddd et al. 2019]. Adaptive Doubly Robust (ADR; a.k.a. stabilized one- bandit data) Luedtke and van der Laan 2016 Hadad et al., 2019 is the same as DR but sets wi, = 9, 1/2 (A(t)(((())(()))))) (t))) ADR is unbiased and asymptotically normal for multi-armed bandit logging policies but is biased for mntext-meesurabbee adaptive logging policies, which is the focus of this paper. Finally, note that our proposal CADR takes the same form as DR but with w. o, using our adaptive conditional standard deviation estimators ō in Eq. (3)
step estimator for
'The code can be found at ht https: // wtt.coom//mmao. http:///ww.aaaaoooooooooooooooo.
8

---

Inner contestual
Incar contoatual
Enoar corteatual
fnear contexitaal polsy
F'war contorual Cargut
CALR 55%, Covoee agg
CALA 40% Coveahio
CADR 57% Coversoe
030
0.5
0.23
630 325 630
180
6dc 325
0:00
035
558 6.5 <36 2lg 106
063 als kie 6/ id
115 050 625 360
Coverage
Coverage
contestual
contestual
frem contectual
free contecctual
contred
tanget
Largel
DADI 93% Corerrage
937% Consrage
CADR 57% Coverege
630
021
650
1.0
5dc 025 630 011 3.00
5.0 626 135
1:00
sin X50 1.0 051
1:5 030 025 35h
Caverage
Coveroge
Coverage
corecnnnaatuu target Bellcy
am coneextuol target colicy
tarpet celicy
targer policy
corgtt policy
CAOIA 0SS Conoragg
95% Conareeg
95% Coverone
0.5
4:1
119
035
3.0
020 6.5
DM 92%
IPW
ADR 95% Coverage
MILD
Lerged polsyy
largel puticy
larger polkcy
tergel polay
Largel polsy
CASI Choece ugr
C42R-59% 57% Covernee
0.5
0.50
0:2
025
023
3:3
020
0.0
km
DW 95% Coverage
IPN 95% Coverage
D6 95% Covarage
AIR 90% Coverage
HADA 55% Coverage
Figure 1: Comparison of CADR estimator against DM, IPW, DR, ADR and MRDR w.r.t. 95% confidence interval coverage on 57 OpenML-CC18 datasets and target policies.
4.2 Contextual Bandit Data from Multiclass Classification Data
To construct our data, we turn K-class classification tasks into K-armed contextual bandit prob- lems [Dudik et al.) 2014 Dimakopoulou et al. 2017, Su et al. 2019), which has the benefits of reproducibility using public datasets and being able to make uncontroversial comparisons using actual ground truth data with counterfactuals. We use the public OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18; BSD 3-Clause license) [Bischl et al. [017]. which has datasets that vary in domain, number of observations, number of classes and number of features. Among these, we select the classification datasets which have less than 100 features. This results in 57 classification datasets from OpenML-CC18 used for evaluation and Table I in section E. of the appendix summarizes the characteristics of these datasets.
Each dataset is a collection of pairs of covariates X and labels L E (1...., K}. We transform each dataset the contextual bandit problem as follows. At each round, we draw (t), L(t) uniformly at random with replacement from the dataset. We reveal the context (t) to the agent, and given an arm pull AL)), we draw and return the reward Y()) ~ (V((((()) = [([)),] 1). To generate our data, we set T = 10000 and use the following e-greedy procedure. We pull arms uniformly at random
until each arm has been pulled at least once. Then at each subsequent round t, we fit Q using the data up to that time in the same fashion used for the DM estimator above using decision- trree regressions. We set (t) = arg max a= I...,K Q (a,()))())) and 6, = 0.01 1-1 We then let (a x) / K for # (1) and ge (A,A(() ( xx) = E e// K. That is, with probability c we pull random arm, and otherwise we pull (t())
We then consider four candidate policies to evaluate: (1) "arm 1 non-contextual"; g0(1 .)) 1 and otherwise g"(a (a x) = 0 (note that the meaning of label "1" changes by dataset), (2) "arm non- contextual": g" (2 x) 1 and otherwise g" (a. z) 0, (3) "linear contextual": we sample new
9

---

dataset of size T using uniform exploration policy, then fit Ør as above using linear regression, fix a" = arg me Q xz), and set g (a" ()) = 1 and otherwise g"(a x) = 0, (4) "tree
contextual": same as "linear contextual" but fit Ør using decision-tree regression.
4.3 Results
Tigure[[showe the comparison of CADR estimator against DM, IPW, DR, ADR, and MRDR w.r.t coverage, that is, the frequency over 64 replications the 95% confidence interval covering the true Vo. for each of the 57 OpenML-CCI8 datasets and 4 target policies. In each subfigure, each dot represents a dataset, the y-axis corresponds to the coverage of the CADR estimator and the z-axis corresponds to the coverage of one of the baseline estimators. The lines represent one standard emror over the 64 replications. The dot is depicted in blue if for that dataset CADR has significantly (i.e., by more than one standard error) better coverage than the baseline estimator, in red if it has significantly worse coverage, and in black if the difference in coverage of both estimators is within one standard error. Results are averaged over 64 replications that ran for each dataset, estimator, and target policy. The standard errors (computed as the sample standard deviation over the 64 replications divided by v (64) are depicted and the 95% coverage targets are shown in dashed lines. In Figure I1 outcome models for CADR, DM, DR, ADR, and MRDR are fit using linear regression (with default sklearn parameters). In the appendix, we provide additional empirical results where use decision-tree regressions, or where we use the MRDR outcome model for CADR, or where we use cross-fold estimation across time.
Across all of our experiments, we observe that the confidence interval of CADR has better coverage of the ground truth than any other baseline, which can be attributed to its asymptotic normality. The second best estimator in terms of coverage is DR. The advantages of CADR over DR are most pro- nounced when either (a) there is a mismatch between the logging policy and the target policy (e.g., compare the 1st and 2nd rows in Figure | the tree target policy is most similar the logging policy, which also uses trees) or (b) when the outcome model is bad (either due to model misspecification such with linear model on real data due to small sample size).
Conclusions
Adaptive experiments hold great promise for better, more efficient, and even more ethical exper- iments. However, they complicate post-experiment inference, which is a cornerstone of drawing credible conclusions from controlled experiments. We provided here a novel asymptotically normal estimator for policy value and causal effects when data were generated from contextual adaptive experiment, such as a contextual bandit algorithm. This led simple and effective confidence in- tervals given by adding and subtracting multiples of the standard error, making contextual adaptive experiments a more viable option for experimentation in practice.
6 Societal Impact and Limitations
Adaptive experiments hold particular promise in settings where experimentation is costly and/or dangerous, such as in medicine and policymaking. By adapting treatment allocation, harmful in- terventions can be avoided, outcomes for study participants improved, and smaller studies enabled. Being able draw credible conclusions from such experiments make them viable replacements for classic randomized trials. Our confidence intervals offer one way to do so. At the same time, and especially subject our assumption of vanishing but nonzero exploration, these experiments must be subject to the same ethical guidelines as classic randomized experiments. Additionally, the usual caveats of frequentist confidence intervals hold here, such as its interpretation only as a guaran- tee over data collection, this guarantee only being approximate in finite samples when we rely on asymptotic normality, and the risks of multiple comparisons and of p-hacking. Finally, we note that our inference focused on an average quantity, as such it focuses on social welfare and need not cap- ture the risk to individuals or groups. Subgroup analyses may therefore be helpful complementing the analysis; these can be conducted by setting g" (a xr) to zero for some x s. Future work may be necessary to further extend our results to conducting inference on risk metries such as quantiles of outcomes.
10

---

References
Susan Athey, Sarah Baird, Julian Jamison, Craig McIntosh, Berk Özler, and Dohbit Sama. A sequential and adaptive experiment to increase the uptake of long-acting reversible contraceptives in cameroon, 2018. URL http://www................rorrar
wor. worldbank.
pdf Study protocol.
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. Ae: domain-agnostic platform for adaptive experimentation. In Workshop System for ML, 2018.
Alina Beygelzimer and John Langford. The offset tree for learning with partial labels. In Pro- ceedings of the 15th ACM SIGKDD international conference Knowledge discovery and data mining, pages 129- 138, 2009.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G Man- tovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint TT4V::770.00777 2017.
Jack Bowden and Lorenzo Trippa. Unbiased estimation for response adaptive clinical trials. Statis- tical methods in medical research, 26(5):2376-23 2388, 2017.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. The Econometrics Journal, 21(1):C1-668 2018.
Maria Dinakopoulou, Zhengyuan Zhou, Susan Athey, and Guido Imbens. Estimation considerations in contextual bandits. arXiv preprint arXiv: 2017.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on International Conference Machine Learning, pages 1097-1104, 2011.
Miroslav Dudík, Dumitru Erhan, John Langford, Lihong Li, et al. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-511 2014.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off- policy evaluation. In International Conference on Machine Learning, pages 1447-1456. PMLR, 2018.
Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence inter- vals for policy evaluation in adaptive experiments. arXiv preprint 1xX.v:191..02768 2019.
Nathan Kallus. Balanced policy evaluation and learning. In Advances in Neural Information Pro- cessing Systems, pages 8895-8906, 2018.
Nathan Kallus and Madeleine Udell. Dynamic assortment personalization in high dimensions. Op- erations Research, 6824))1020-1037, 2020.
Nathan Kallus and Masatoshi Uchara. Intrinsically efficient, stable, and bounded off-policy evalua- tion for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Maximilian Kasy and Anja Sautmann. Adaptive treatment assignment in experiments for policy choice. Econometrica, 19-1321 2021.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. contextual banddt approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.
Lihong Li, Wei Chu, John I Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. Proceedings of the fourth ACM inter- national conference Web search and data mining, pages 297-306, 2011.
11

---

Alexander R. Luedtke and Mark J. van der Laan. Statistical inference for the mean outcome under possibly non-unique optimal treatment strategy. The Annals of Statistics, 44(2):713 742, 2016. doi: 10. 2214/15-AOS1388
1214/15-A0S1361
Xinkun Nie, Xiaoying Tian, Jonathan Taylor, and James Zou. Why adaptively collected data have negative bias and how to correct for it. In International Conference on Artificial Intelligence and Statistics,, pages 1261-1269. PMLR, 2018.
Sheng Qiang and Mohsen Bayati. Dynamic pricing with demand covariates. arXiv preprint arXiv:1604.07465 2016.
Simon Quinn, Alex Teytelboym, Maximilian Kasy, Grant Gordon, and Stefano Caria. A sequen- tial and adaptive experiment increase the uptake of long-acting reversible contraceptives in cameroon, 2019. URL httpps:/ww...ccaalsseennnggggggs registration.
Study
James Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89 (427))846-866, 1994.
Jachyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. On the bias, risk and consistency of sample means in multi-armed bandits. arXiv preprint arXiv:/902.00746 2019.
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning, pages 6005-6014. PMLR, 2019.
Ambuj Tewari and Susan A Murphy. From ads interventions: Contextual bandits in mobile health. In Mobile Health, pages 495-517. Springer, 2017.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
Mark J van der Laan and James M Robins. Unified methods for censored longitudinal data and causality. Springer Science & Business Media, 2003.
A. der Vaart and Wellner. Weak Convergence and Empirical Processes. Springer- New York, 03 1996. ISBN 9781475725452.
Aad W van der Vaart. Asymptotic statistics. Cambridge university press, 2000.
Yu- fu-Xiang Wang, Alekh Agarwal, and Miroslav Dudık. Optimal and adaptive off- off-policy evaluation in contextual bandits. In International Conference on Machine Learning, pages 3589-3597. PMLR, 2017.
Min Xu, Tao Qin, and Tie Yan Liu. Estimation bias in multi-armed bandit algorithms for search advertising. Advances in Neural Information Processing Systems, 26:2400-2444 2408,, 2013.
Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. arXiv preprint vrXiv:2106.02022 2021.
Kelly W Zhang, Lucas Janson, and Susan A Murphy. Statistical inference with m-estimators on adaptively collected data. arXiv preprint arXiv:2104.14074 2021.
12


Reviews:
[{"review_id": "z6165HufpIw", "reviewer": "Reviewer_hz9e", "summary": "This paper is motivated by the fact that no asymptotically normal estimators exist when the data is collected context-dependent adaptively. This work provides such estimators (named CADR, Algo. 1) exhibiting asymptotic normality (Thm. 1). ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is motivated by the fact that no asymptotically normal estimators exist when the data is collected context-dependent adaptively. This work provides such estimators (named CADR, Algo. 1) exhibiting asymptotic normality (Thm. 1). \n\nThis paper is based on the solid theoretical foundation and sophisticated observation that asymptotically normal estimators have only been developed under a non-contextual adaptive setting. This observation came as a surprise to me because the contextual adaptive setting is quite practical, but we still don't have a good estimator on it. Theories are strong, and experiments corroborate with the theory. \n\nHere are some points that I'd like to ask for authors to help readers' understanding a bit more. \n* C1. Line 30-32 — \"However, due to the adaptive nature of the data collection, unlike classic randomized trials, standard estimates and their confidence intervals actually fail to provide correct coverage\" seems to be a key motivation of the paper. I think it should be more enlightened to make readers motivated.\n\n* C2. Line 62-72. It would be helpful if you could provide a graphical representation (i.e., DAG) for encoding the assumption, such as \"Xt is independent of all else given A(t).\" This would help to be more explicit and automatically tell some questions such as \"What is all-else in the assumption\"? \n\n* C3. Line 110-117. Can you formalize this observation with some examples? Despite the importance of this paragraph (this paragraph gives the very motivation of this work), explanations seem weak. \n\nI have one question about this work. \n* Q1. In Algo. 1, did you use sample-splitting (or cross-fitting) technique? If not, does the result still hold? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Assumptions are written in explicit and transparent manner. Still, I think it'd be more helpful if authors can provide a DAG representation for encoding the dependency assumption in Line 62-72. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "qAFAz6jyRr", "reviewer": "Reviewer_hHgb", "summary": "This paper presents a method for estimating the value of some fixed evaluation policy of interest using data that was collected with a contextual bandit algorithm. They prove that their estimator is asymptotically normal even when the model of the expected reward conditional on state and action is misspecified.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The result the authors present is interesting and appears to be an important contribution to the literature particularly because the model of the reward model can be misspecified. However, I believe that the paper is currently still weak because (1) insufficient discussion of the relationship to previous literature and (2) limited empirical evaluation. I would be willing to raise my score if the authors improve in these two respects.\n\nRegarding previous literature\n-\tI am still unclear about how exactly your method differs from that of Hadad et al., 2019 and I think there should be additional discussion. In a non-contextual setting, is your estimator equivalent to that of Hadad et al. in a special case? How is your weighting scheme similar / different from that of Hadad et. al, 2019? There should be a more explicit comparison to understand how your contribution is different.\n-\tLine 127-128: This comment is vague. Could you elaborate on the relationship to Hadad et al., 2019. Later at Line 256, you say the main focus of the paper is that the estimator of Hadad is biased on contextual bandit data; this should be stated earlier if this is the case. And more discussion should be made, is your proposed estimator unbiased on contextual bandit data? What exactly is your argument that Hadad et. al, 2019 is insufficient for contextual bandit data?\n-\tHow is your work related to more recent related work:\no\tOff Policy Evaluation via Adaptive Weighting with Data Collected from Contextual Bandits: https://arxiv.org/abs/2106.02029 \no\tStatistical Inference with M-Estimators on Adaptively Collected Data: https://arxiv.org/abs/2104.14074 \n\nEmpirical Evaluation\n-\tThe Figure 1 plots are quite difficult to interpret, what do the colors mean? What are the different settings you consider? How do you compute standard errors?\n-\tIn your empirical evaluation you have T=10000, it is unclear how your method performs in smaller sample settings, which is commonly the case in real world problems. \n-\tHow is the empirical performance of your estimator affected if the model Q does not converge to the true value (if at all)?\n-\tAre there any limitations or downsides of your approach empirically due to the large number of terms that must be plugged-in?\n\nAdditional Questions\n-\tLine 80: You define g^*, but in the equation above you use g. Is this intentional?\n-\tTheoretically / in practice do we expect Assumption 4 to hold?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "os3MD_Bo9H3", "reviewer": "Reviewer_2WyM", "summary": "This study proposes a variance-stabilized estimator for policy evaluation from data obtained from contextual bandit algorithms, which is inspired by [Luedtke and van der Laan 2016]  and [Hadad et al. 2019].  In this study, the authors estimate the variance of the score function (canonical gradient) from samples, which standardizes the MDS for applying martingale CLT.\n[Luedtke, and van der Laan 2016] Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy  \n[Hadad et al 2019] Confidence Intervals for Policy Evaluation in Adaptive Experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I vote for clear reject.  \nI appreciate that this problem tackles a mathematically challenging problem.  \nAlso, there is not enough of a survey of the bandit problem, and it is not clear under what circumstances the proposed method would be needed.  \nAlthough the authors do not mention it much, several methods of statistical inference from contextual moon bandits have already been proposed.  \nCompared with them, I consider that the authors' method aims to enable statistical inference in more difficult situations.  \nHowever, for me, it seems that it is unnecessary for the bandit problem, that is, there is a few concrete examples of a bandit algorithm in which the proposed method is needed.  \nFurthermore, Assumptions 5 and 6 restrict the logging policy (sampling strategy of bandit algorithm), and it is not clear what specific bandit algorithms satisfy these assumptions.  \nIn addition, the essential contribution of this paper seems to be the proof of Theorem 2 (other parts have already been known).   \nHowever, the notation is complicated, and there are some omissions and typos, so I could not check it sufficiently.   \nThe inability to read the proofs is partly my problem and partly due to the time constraints of the conference review process, but I also think that the authors could make an effort to make the proof more readable.  \nIn order to increase the score, the authors need to \n- explain the connection with existing studies of the \"bandit problem\" or change the field that the authors want to contribute.\n- make proofs and notations easier to read.  \nDetailed comments are listed below.\n\n# Major concerns\nI have three major concerns for this paper: \n- Point 1: lack of connection to bandit problem and the usefulness of the proposed methods.\n- Point 2: theoretical correctness and implicit restriction to the bandit algorithms;\n- Point 3: the setting of the experiment may not be appropriate.\nAmong them, I am particularly concerned with Point 1, that is, the proposed method is not designed for the bandit problem.\n\n## Point 1\n### Existing studies\nThe paper does not cite much of the bandit literature, and it is not clear how the proposed method can be applied.  \nIn addition, there are also several studies on evaluation from data gathered from adaptive experiments (including bandit algorithms) although not cited in this paper.  \nFor example,  \n[van der Laan 2008] The Construction and Analysis of Adaptive Group Sequential Designs  \n[Hahn et al 2011] Adaptive Experimental Design Using the Propensity Score  \nI think that the following papers also propose such methods.  \n[van der Laan and Lendle] Online Targeted Learning   \n[Chambaz et al. 2014] Targeted Covariate-Adjusted Response-Adaptive LASSO-Based Randomized Controlled Trials  \n[Tabord-Meehan 2020] Stratification Trees for Adaptive Randomization in Randomized Controlled Trials  \nI can list other more studies tackling this problem.  \n\nSome of these studies assumes that the contextual policy $g_t(a | x)$ converges to a time-independent measure and $g_t(a| x)\\neq 0$.  \nThis is a natural assumption for some bandit problems (but not for regret minimization).    \nUnder this assumption, we can obtain asymptotically normal estimator with a simpler form, such as the IPW (IPS) estimator without the variance-stabilization.\n\nIn situations where the data is in batch form, statistical inference also can be performed even if the policy does not converge.  \n[Zhang et al 2020] Inference for Batched Bandits  \nThis is essentially a generalized method of moments (GMM).\n(Although it is not well known, a GMM estimator is known to have asymptotic normality for martingales)\n\nCompared to these uncited previous studies, I cannot understand how useful the proposed method is to the bandit community.  \nIn addition, as described below, I cannot think of a specific bandit case where this proposed method would be necessary.  \n \n### Variance-stabilized estimators and fewer applications in bandit problem\nI consider that the contribution of this study lies in avoiding sample splitting, which (I think) is used in [Luedtke, and van der Laan 2016] and [Kato 2020], to obtain $\\sqrt{N}$-consistent estimator.  \n[Kato 2020] Confidence Interval for Off-Policy Evaluation from Dependent Samples via Bandit Algorithm: Approach from Standardized Martingales  \nBy using sample splitting, we can show asymptotic normality without making assumptions about the contextual policy.   \nInstead, with that sample split, it is no longer $\\sqrt{N}$-consistent.  \nRecently, [Hadad et al 2019] proposed a method for statistical inference from a bandit data without context, which attains  $\\sqrt{N}$-consistency.  \nIt can avoid estimation of the variance by considering bandit problem without context.  \nIn this study, the authors do not use sample-splitting and obtain a  $\\sqrt{N}$-consistent estimator.  \n\nHowever, I believe that these methods are essentially unnecessary for the bandit problem, and a simpler estimator is sufficient.\nThe reasons for this are listed below.\n- If the logging policy converges, asymptotic normality can be obtained using other simpler estimators, such as IPW (IPS) estimators without using the methods of [Hadad et al 2019] or [Luedtke, and van der Laan 2016].  \n- These papers presume stochastic bandit problem, but it is difficult to find a bandit algorithm, where the logging policy does not converge. \n- None of [Hadad et al 2019] or [Luedtke, and van der Laan 2016] can be used for the regret minimization, where the logging policy converges to zero. (In this case, the cannonical gradient diverges).\n- In bandit problems, the target policy (denoted by $g^*$ in this paper) to be evaluated is also time-dependent, and there is no problem setting in which a time-independent target policy exists in advance and it is evaluated.  \nI also think that [Luedtke, and van der Laan 2016] and [Hadad et al 2019] consider a different situation from bandit problem.  \n\n### Lack of connection to bandit problem\nI explain this in more details.  As well as [Luedtke, and van der Laan 2016] and [Hadad et al 2019],  the proposed method is needed if a bandit problem satisfies the following situations:\n1. Bandit model is stochastic bandit (so not adversarial bandit);\n2. The goals is not regret minimization (logging policy does not converges to $0$ or take $0$ value);\n3. The policy (sapling strategy) does not converge;  \n\nIf not 1., we cannot assume the data generating process and define the causal effect.  \nIf the bandit algorithm aims to minimize the regret, the canonical gradient will diverge.  \n(Variance also becomes $\\infty$. Exclusion of regret minimization is Asssumtpion 6?)   \nThe authors raised a problem that \"IS ratios can both diverge to infinity or converges to zero\" in line 114.   \nIf the IS ratios really diverges, we cannot define the variance.  \nTo avoid this problem, we need to carefully restrict the class of the logging policy.  \nFor instance, [Hadad et al 2019] also restricted the class in their Assumptions 1--3.  \nIn this paper, it seems that Assumption 5 corresponds to the  Assumptions 1--3 of [Hadad et al 2019].  \nCan the authors show the method of Li et al 2010, Kallus and Udell 2020, Qiang and Bayati 2016 etc, which the authors cited in Introduction, satisfy these conditions?    \nIn contrast, if the sampling strategy converges, we do not have to use variance stabilization.   \nFor instance, the IPW (IPS) estimator $y/ g(a| x)$ also have asymptotic normality owing to the martingale property.  \n(In this paper $\\hat{\\Psi}^{IPS}$ without variance stabilization. Without variance stabilization, this type of estimator is often used.)\n\nIn addition,   \n 4. The authors also seem to restrict the sampling strategy (logging policy) in Assumption 5.  \nThis is a specific assumption of this paper to show Theorem 2.  \nWhat kind of bandit algorithms satisfy this assumption?   \nIn bandit literature, this kind of assumption is carefully discussed in the paper.  \nFor instance, the following paper considers a nonparametric method for regret minimization with carefully restricting the class of sampling strategy.   \n[Yang and Zhu 2002] Randomized Allocation with nonparametric estimation for a multi-armed bandit problem with covariates  \nThus, I think that Assumption 5 (and 6) is not trivial, and the authors should discuss more carefully.  \n\nIn conclusion, I do not know whether there is a bandit problem where the proposed method is needed.  \nThis is not to deny the author's attempt, but if the author thinks the proposed method is necessary for bandit problem, I would like to know the specific bandit paper.  \nThe paper cites some papers in the field of adaptive experimental design and causal inference, but not many of papers in bandit problem, which is main topic of this paper.   \nThere are several cited papers, but I think that all of them converge on the sampling strategy or do not satisfy Assumption 5--6,.    \nThat is, the authors should motivate the proposed method more from the viewpoint of bandit algorithms.    \nOtherwise, the authors should change the field that they want to contribute.   \nI believe that the current structure of the paper, if accepted, will confuse the bandit community.  \n\n## Point 2:\n### Assumption 5 and 6\nAs pointed in Point 1, Assumption 5 and 6 restrict the logging policy. \nThese assumptions should be more paid attention because it is closely related to design of the bandit algorithm. For example, [Yang and Zhu 2002] more carefully discuss the assumption.  \nIn addition, the authors should cite an example of bandit algorithm satisfying these assumptions.  \n(But the logging policy does not converge. As mentioned above, if it converges, we do not have to use the proposed method)  \n\n### Proof of Theorem 2\nI think that Theorem 2 is the most important in this paper.\nBecause if we use sample splitting, we can show the asymptotic normality more easily by losing the $\\sqrt{T}$-consistency for a sample size $N$ ([Luedtke, and van der Laan 2016]).  \nHowever, owing to the difficulty of the notations, we cannot fully understand the proof.  \nCould the authors answer the following questions?\n- Does line (6) of $\\ell 571$ in supplementary material is MDS?\nThe authors may omit $g_t$, but I think $g_t$ is correlated with past samples. Does it satisfy MDS?\n- Do we need to assume boundedness or sub-gaussian to the outcome $Y$?\n- If we do sample splitting, can't we prove asymptotic normality without assuming Assumptions 5 and 6, instead of losing $\\sqrt{N}$-consistency? [Luedtke, and van der Laan 2016] and [Kato 2020] (I think this paper is essentially same as [Luedtke, and van der Laan 2016]) do this?   \nIf sample splitting can solve the problem, the authors should explain this fact even if they are inferior to the proposed method in the sense of $\\sqrt{N}$-consistency?  \n\n## Part 3\nIn experiments, the logging policy seems to converge.  \nIf so, we do not have to use the proposed variance stabilization and obtain asymptotic normality with simpler estimators proposed in existing studies.  \nIn addition, If so, the authors should change the setting of experiments.  \n\n# Minor points\n- As pointed in Part 1, there are many methods for causal inference from contextual experimental designs in situations where the logging policy converges, but they are almost never cited.\n- I am a bit familiar with this notation, so I could read it, but this notation would be very difficult for machine learning people to read. More explanation of terminology is needed, since prior knowledge differs between the bandit and causal inference communities.\n- The notation in this paper is difficult and the proof is very hard to read. I don't mind that the proof is difficult, but there are many notations that are not explicitly defined, making it difficult to read. In addition, there are typos in some places.  \n- Assumption 5 constrains the behavior of the original Bandit algorithm. However, if we are going to put constraints on the algorithm, why not design the algorithm so that it is easy to evaluate measures?   \n\n# Requirements to the author\nI would require the authors to answer the following point\n- Raise an example of a situation of bandit problems, where this algorithm is needed, with justifying Assumption 5 and 6;\nand improve following points in the next revision:\n- Reorganization of Notation;\n- Another experiment using  logging policy that needs the proposed method;\n- A better literature review.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not describe a specific bandit algorithm, where the proposed method can be applied.   \nAs pointed in Main review, there are three possible settings for the bandit problem settings:\n- regret minimization: Cannot be applied to regret minimization where the variance of the IPW part jumps infinitely (Assumption 6?).\n- best arm identification: There is no generally accepted research on contextual best arm identification.\n- adversarial bandit: This is not the data generating process considered in this paper.\nI do not believe the proposed method can be used for any of these.  \n\nIn addition, for example, in a problem where the logging policy converges and does not take $0$ value, the IPW (IS) estimator is sufficient without the proposed method.   \nThe authors state that IPW (IS) estimators \"are (not) generally asymptotically normal under adaptive data collection\" (line 253), but this is not true. (line 253) because if the logging policy converges, the IPW estimator will asymptotically follow normality by the martingale CLT.   \nThe authors also pointed out the problem that the logging policy take $0$.  \nIndeed, under this situation, the IPW (IS) estimator cannot be used, but the proposed method also cannot be used (Assumption 6).  \nAfter all, the authors have made (implicit) assumptions on the sampling strategy, and they should discuss whether those assumptions are more valid than other assumptions.\n\nMoreover, in the setting of the experiment, it appears that the authors consider a problem where the logging policy converges.  \nSo, I think that the authors do not have to use their proposed method to obtain asymptotic normal estimator.  \nIf my comment is incorrect, the authors should clarify this in the paper.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "More than 20 hours"}, "extra_text": {}}, {"review_id": "ET8mG-A_FmZ", "reviewer": "Reviewer_SpRC", "summary": "The current paper concerns inference of quantities such as\ntreatment effects or policy values from data that is collected\nfrom contextual bandit experiments, Motivated by applications\nin healthcare, revenue management or product recommendations\nwhere randomized experimentation is not often the norm.\nThey isolate a key issue (variance stabilization), identified\nin previous works on analysis of adaptively collected data (see\npaper for references), and propose estimators for average causal effects of the \n`doubly robust` form to overcome this issue. The main result\nis a central limit theorem for this estimator, allowing\nfor frequentist inference (i.e. confidence intervals, p values) in the standard way.\nThe estimator relies heavily on a sequence of conditional variance estimators, which \nthey construct using a judicious use of averaging and a base regression model for the outcome (\nwhich may be misspecified even in the limit).\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem dealt with in the paper is well-motivated and of significant interest\nto the NeurIPS community. The paper is, modulo some suggestions below, well-written. \nI would advocate its inclusion in the conference program once the major comments\nbelow are addressed.\n\nMajor comments:\n\n1. Assumption 6 supersedes Assumption 3, saying the exploration rate needs to be\nsignificantly greater than the worst case 1/sqrt(t) rate. This is crucial to the\nconstruction of the CADR estimator and it is somewhat disingenuous to isolate it thus, unless\nyou have concrete scenarios where an alternative estimation scheme might exist that\ndoes not require this exploration rate.\n\n2. In the same vein as point 1, I don't think the form of Assumption 3 is sufficient for\nthe Lindeberg condition argument as laid out. For instance, consider the array Z_{t, T} = \\sqrt{t/T} \\eps_t\nwhere \\eps_t are i.i.d. random signs. This satisfies Assumption 3 but not the Lindeberg condition.\nPossibly the argument may be tightened, but at the moment, it seems you need that the exploration is\n\\Omega(1/sqrt(t)).\n\nModerate comments:\n\n1. Can you say how your CADR estimator reduces to the variance stabilizing construction in Hadad et al 2019 and, if not, what are the\ndifferences? This would aid intuition for the reader.\n\n2. Can you comment on the efficiency of the constructed estimator, or if we might need to look outside this class of estimators\nfor the most efficient estimator? Alternatively, a simple setting where it is the most efficient?\n\n3. Some popular policies like UCB are not covered by your scheme given they have trivial propensities, it is good \nto mention this. I would also welcome a remark on how Thompson sampling type policies might apply.\n\nMinor comments:\n\n1. L.80 ambiguity with g vs g^*\n2. Assumption 5 needs to be restated; perhaps missing punctuation/sentence splitting.\n3. Similar ambiguity with Q_{0, X}, Q_{0,Y} vs Q_X, Q_Y in L78, L79. In general, the `0` subscript is a little overused. \n4. Define \\cal{O} as the space of a, x, y triples (I assumed this and faced no obvious errors)\n5. The use of a \\bar{} is a bit inconsistent: \\bar{O}(t), \\bar{Q} and \\bar{\\Phi} mean different things (a sequence, \nan empirical average and just another thing). This causes some unnecessary readability issues.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors' work allows to use observational data from contextual bandit experiments for statistical inference purposes. In principle this can have potential societal impact, but would be within the context of a specific application (e.g. personalized medicine, policy making...). The authors' conclusion makes some aspects of this apparent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you to the authors and the reviewers for their contributions! There was a long discussion on this paper, and we sought external opinions to add clarifications. The final consensus is that the paper makes an interesting technical contribution to the literature on inference over adaptively collected data, specifically via contextual bandits. However, the paper is not very clear in terms of its key results and their limitations, and I urge the authors to clarify the exposition in the next version. Key issues have already been noted in the discussion period; one reviewer also specifically requested clarifications on whether Assumptions 5 and 6 hold in the experiments (i.e., whether the \epsilon is decaying too fast) or to adapt the experiment parameters accordingly.
Decision: Accept (Poster)