Statistical Inference with M-Estimators on Adaptively Collected Data
Authors: Kelly Wang Zhang
Lucas Janson
Susan Murphy
Venue: NeurIPS 2021 Poster (2021)
Keywords: contextual bandit algorithms
confidence intervals
adaptively collected data
causal inference

Abstract: Bandit algorithms are increasingly used in real-world sequential decision-making problems. Associated with this is an increased desire to be able to use the resulting datasets to answer scientific questions like: Did one type of ad lead to more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail to provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is a lack of general methods for  conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward. In this  work, we develop theory justifying the use of M-estimators---which  includes estimators based on empirical risk minimization as well as maximum likelihood---on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators, modified with particular adaptive weights, can be used  to construct asymptotically valid confidence regions for a variety of inferential targets.

Statistical Inference with M-Estimators on Adaptively Collected Data
Kelly W. Zhang
Lucas Janson
Department of Computer Science
Departments of Statistics
Harvard University
Harvard University
ke kellywzhangdseas. harvard.ede
1 jjansonffas www.aar. edu
Susan A. Murphy
Departments of Statistics and Computer Science
Harvard University samurphy@fas harvard. edu
Abstract
Bandit algorithms are increasingly used in real-world sequential decision decision-making problems. Associated with this is an increased desire to able to use the resulting datasets to answer scientific questions like: Did one type of ad lead more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is lack of general methods for conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in logistic regression model for binary reward. this work, we develop theory justifying the use of M-estimators which includes estimators based on empirical risk minimization as well as maximum likelihood on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators, modified with particular adaptive weights, can used to construct asymptotically valid confidence regions for variety of inferential targets.
Introduction
Due to the need for interventions that are personalized to users, (contextual) bandit algorithms are increasingly used address sequential decision making problems in health-care [Yom-Tov al.
2017, et al. (020). online education [Liu et al. 2014 Shaikh et al.) 2019), and public policy
Kasy and Sautmann 202T Caria et al. 2020 Contextual bandits personalize, that is, minimize regret, by learning to choose the best intervention in each context, i.e., the action that leads to the greatest expected reward. Besides the goal of regret minimization, another critical goal in these real-world problems is to be able to use the resulting data collected by bandit algorithms to advance mean
scientific knowledge [Liu al.) 2014 Erraqabi al. 2017). By scientific knowledge, information gained by using the data to conduct variety of statistical analyses, including confidence interval construction and hypothesis testing. While regret minimization is wethinn-eperrmmet learning objective, gaining scientific knowledge from the resulting adaptively collected data is a between-experiment learning objective, which ultimately helps with regret minimization between deployments of bandit algorithms. Note that the data collected by bandit algorithms are adaptively collected because previously observed contexts, actions, and rewards are used to inform what actions to select in future timesteps.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

There are a variety of between-experiment learning questions encountered in real-life applications of bandit algorithms. For example, in real-life sequential decision- makking problems there are often a number of additional scientifically interesting outcomes besides the reward that are collected during the experiment. In the online advertising setting, the reward might be whether an ad is clicked on, but one may be interested in the outcome of amount of money spent or the subsequent time spent on the advertiser's 's website. If it was found that an ad had high click-through rate, but low amounts of money was spent after clicking the ad, one may redesign the reward used in the next bandit experiment. One type of statistical analysis would be to construct confidence intervals for the relative effect of the actions on multiple outcomes (in addition to the reward) conditional on the context. Furthermore, due to engineering and practical limitations, some of the variables that might be useful as context are often not accessible to the bandit algorithm online. If after-study analyses find some such contextual variables to have sufficiently strong influence on the relative usefulness of an action, this might lead investigators ensure these variables are accessible the bandit algorithm in the next experiment.
As discussed above, we can gain scientific knowledge from data collected with (contextual) ban- dit algorithms by constructing confidence intervals and performing hypothesis tests for unknown quantities such as the expected outcome for different actions in various contexts. Unfortunately, standard statistical methods developed for i.i.d. data fail to provide valid inference when applied to data collected with common bandit algorithms. For example, assuming the sample mean of rewards for arm is approximately normal can lead to unreliable confidence intervals and inflated type- 1 error; see Section 3.T for an illustration. Recently statistical inference methods have been developed for data collected using bandit algorithms (Hadad et al. 2019 Deshpande et al.] 2018 Zhang ct al. 2020]; however, these methods are limited to inference for parameters of simple models. There is a lack of general statistical inference methods for data collected with (contextual) bandit algorithms in more complex data-analytic settings, including parameters in non-linear models for outcomes; for example, there are currently no methods for constructing valid confidence intervals for the parameters of a logistic regression model for binary outcomes or for constructing confidence intervals based on robust estimators like minimizers of the Huber loss function.
In this work we show that a wide variety of estimators which are frequently used both in science and industry i.i.d. data, namely, M-estimators (Van der Vaart 2000]. can be used to conduct valid inference on data collected with (contextual) bandit algorithms when adjusted with particular adaptive weights, i.e., weights that are function of previously collected data. Different forms of adaptive weights are used by existing methods for simple models (Deshpande et al. 2018 Hadad et al., 2019 Zhang et al., 2020). Our work is a step towards developing a general framework for statistical inference on data collected with adaptive algorithms, including (contextual) bandit algorithms.
2 Problem Formulation
We assume that the data have after running contextual bandit algorithm is comprised of contexts {X,)== actions { and primary outcomes {Y.) T is deterministic and known. We assume that rewards are deterministic function of the primary outcomes, i.e., RS f(;) for some known function f. We are interested in constructing confidence regions for the parameters of the conditional distribution of Y, given Ar). Below we consider T - do in order to derive the asymptotic distributions of estimators and construct asymptotically valid confidence intervals. We allow the action space A to be finite or infinite. We use potential outcome notation [mbens and Rubin 2015) and let (a) a E A} denote the potential outcomes of the primary outcome and let Ye
(A,) be the observed outcome. We assume stochastic contextual
E A} i.dd P E for t [1: T); the contextual bandit environment distribution P is in a space of possible environment distributions P. We define the history H. = (Xe, Ar, Ye Y for / > 1 and Ho == 0. Actions A, E A are selected according policies π = [m) |22 which define action selection probabilities Iee (A,, X,, H4-1) == TP (A, 14411 1, X2). Even though the potential outcomes are i.i.d., the observed data {Xe, At, Yt) are not because the actions selected using policies n. which are function of past data, H 1--1.17 Non-independence of observations is a key property of adaptively collected data.
bandit environment in which (X,, (a):
We are interested in constructing confidence regions for some unknown 0" (P) E C Rd, which is parameter of the conditional distribution of Y, given (X,, At). This work focuses on the setting in which we have well-specified model for Y,. Specifically, we assume that 0* (P) is conditionally
2

---

maximizing value of criterion mø, i.e., for all P E P.
0 (P) E argmax Ep [mo A,) w.p. 1.
(1)
BEB
Note that 0* (P) does not depend on (X,, A,) and it is implicit modelling assumption that such 8" (P) exists for a given ma. Note that this formulation includes semi-parametric models, e-g., the model could constrain the conditional mean of Y, be linear in some function of the actions and context, but allow the residuals to follow any mean-zero distribution, including ones that depend on the actions and/or contexts.
To estimate 8* (P). we build on M-estimation Huber 1992 which classically selects the estimator to be the 0 E θ that maximizes the empirical analogue of Equation T
1 mY, At).
Ōr == argmax OE
For example, in a classical linear regression setting with |A| < oo actions, natural choice for my is the negative of the squared loss function, X,, Ar) = - (Y; x, 0a.)². When Y; is binary, natural choice is instead the negative log-likelihood function for a logistic regression model, i... mo (Y,, Xx, = - -Y,,X, 0Λ.- - log(1 + exp(X, 0A.))] More generally, my is commonly chosen to be log- likeliiokoood function the negative of robust loss function such as the Huber loss. If the data, (X,, A,, Y were independent across time, classical approaches could be used to prove the consistency and asymptotic normality of M-estimators [Van der Vaart 2000)_ However,
data collected with bandit algorithms, standard M-estimators like the ordinary least-squares estimator fail provide valid confidence intervals (Hadad et al. 2019 Deshpande al. 2018 Zhang et al. 2020). In this work, we show that M-estimators can still be used to provide valid statistical inference on adaptively collected data when adjusted with well-chosen adaptive weights.
3 Adaptively Weighted M-Estimators
We consider a weighted M-estimating criteria with adaptive weights We E (e, given
by We = V x² (.).)...
Xx, Here (x1t Jr21 (12) are pre-specified stubilizing policies that do not depend default choice for the stabilizing policy when the action space is of size
on data (Y, A,,
|A| < 00 is just The (a. x) 1/|_A| for all x, a, and t; we discuss considerations for the choice of
(me (T [== in Section 3.3 We call these weights square-rool importance weights because they are the
square-root of the standard importance weights (Hammersley] 2013, Wang et al., 2017). Our proposed
estimator for 0* (')P), 07, is the maximizer of weighted version of the M-estimation criterion of Equation
ÔT = argmax T E Womo X1, A.) =: argmnax Mr(().
dee
Note that (0) defined above depends on both the data (Xe, Ar, Y.)T and weights (Wi)T--- We provide asymptotically valid confidence regions for 0" (P) by deriving the asymptotic distribution of o as T + oo and by proving that the convergence in distribution is uniform over P e P. Such convergence allows us to construct a uniformly asymptotically valid - a level confidence region, (a,),, for At (P), which is confidence region that satisfies
inm inf Pr. (P* E Cr(o)) T- PEP
If Cr( were not uniformly valid, then there would exist an c > 0 such that for every sample size T. Cr(a''s coverage would be below a for some worst-case Pr E P. Confidence regions which are asymptotically valid, but not uniformly asymptotically valid, fail to be reliable in practice [Leeb) and Pötscher 2005 Romano et al. 2012). Note that on i.i.d. data it is generally straightforward to show that estimators that converge in distribution do so uniformly; however, as discussed in Zhang
et al. (2220) and Appendix D this not the case on data collected with bandit algorithms.
To construct uniformly valid confidence regions for 0 (P)) we prove that O is uniformly asymptoti-
cally normal in the following sense:
Er(P) 1/2 (r( 0² (P)) 2 N (0, uniformly over P P.
(P)
(4)
3

---

where Mr(0)= o²
(P)((, Xx, A.) We define ring ag mg. Similarly we define respectively my and mg the second and third partial derivatives of mg with respect to 0. For any vector we define 202 = =2
अhg Mr(() and Er(P) + Ei=Be xx (p)
3.1 Intuition for Square-Root Importance Weights
The critical role of the square-root importance weights W, (e is to adjust for instability in the variance of M-estimators due to the bandit algorithm. These weights act akin to standard importance weights when squared and adjust key term in the variance of M-estimators from depending on adaptive policies { which can be ill ill-behaved, to depending on the pre-
TT' (Ae,
specified stabilizing policies {xz² Jt=1- See Zhang al. 2020 and Deshpande al. (2018) for more
discussion of the ill-behavior of the action selection probabilities for common bandit algorithms,
which occurs particularly when there is no unique optimal policy.
As an illustrative example, consider the least-squares estimators in a finite-arm linear contextual bandit setting. Assume that Ep(Yi A, a. X (P) w.p. 1. We focus on estimating (P) for some a E A. The least-squares estimator corresponds to an M-estimator with me, (Y,, Xx, A₂) =
- (Y= - x,T0.)² 0.)2. The adaptively weighted least-squares (AW-LS) estimator is f.a argmax,, 8_ {- ΣT WellA,aaY, - xT) For simplicity, suppose that the stabilizing policy does not change with t and drop the index to get -. Taking the derivative of this criterion, we get 0 = E'= W, X,C, X, (T and rearranging terms gives
T
1
Wi A
(BAWAS
((())
Σ VT
1
W, 1l A. (Yc
X, (T()) (5)
Note that the right hand side of Equation
55 is martingale difference sequence with respect to for all 1; by law of iterated
history (He Y because Ep,x(W,,,. (W,,Aa (Y. x, (P))
expectations and since W. (14-1,1 X,, A,), Ep. W. (Y x, (P)) equals
Ep (W X,, Hp Yr x, (2) X,, A: = a| [
Ep [Warela, X., H [Y x, (P) X,, 4, a] H4: (#i)
0.
(i) holds by our i.i.d. potential outcomes assumption. (ii) holds since Ep Yr A, = a| x, (P).. We prove that 5) is uniformly asymptotically normal by applying a martingale central limit theorem (Appendix B.4.. The key condition in this theorem is that the conditional vari- ance converges uniformly, for which it is sufficient to show that the conditional covariance of A.=a (Y,- X, (P)) given H4-1 equals some positive-definite matrix E(P) for every t, i.e.
Ep,x W 1 (Y- x, (P)) 2 [Pa-1] E(P).
(6)
By law of iterated expectations, Ex,,, [W? 1A (Y, xX XTT:(P))² (
He- equals
7 (Ar, Xz)
Ep
Ep,
Te (A Xe, Ho-1) -A (Yi (Y: xx,T / Xr,
H
(7)
Ep, LAi=aXeX, (Y, - x, {T)) 3 He X1
Ep
H
(0)
Ep
Ep, IA (Y, x, xTT)))))² X
Hi
(6)
Ep Ep IAI= (Y--x, ) x. x
(c)
Ep, [1x=aX,X]]] (V, - x, [(())) E(P).
Above, (a) holds because the importance weights change the sampling measure from the adaptive policy Te to the pre-specified stabilizing policy It (b) holds by our i.idd potential outcomes

---

assumption and because T is pre-specified policy. (c) holds because X, does not depend on H4-1 by our i..d. potential outcomes assumption. (d) holds by the law of iterated expectations. Note that E(P) does not depend on / because a is not time-varying. In contrast, without the adaptive weighting, i.c., when W₂- = 1, the conditional covariance of A,=a (Y; x, 0- (2)) on H4-1 is random variable, due the adaptive policy πT.
In Figure Te plot the empirical distributions the z-statistic for the least-squares estimator both with and without adaptive weighting. We consider a two-armed bandit with A, {0, 1}. Let (P) == (1]))(1)) and ma, (Y, A,) = AA 0,) The unweighted version, i..., the ordinary least-squares (OLS) estimator, is GOLS = argmaxØ, T T.
1 Et=i ma, (Y,, At). The adaptively weighted
version is d == argmaXo,, + Σ Weme, At). We collect data using Thompson Sampling and use uniform stabilizing policy where (1) π -h (0) 0.5. It is clear that the least- estimator with adaptive weighting has a z-statistic that is much closer to normal distribution.
Distribution of OLS Estimator
Distribution of AW-Least Squares Estimator
0..
0.4
0.3
0.3-
lype-1 %
Type-1 error:
standard normel Standard Normal Fmpirial Dishibutioon
0.2
9.6 %
0.1
0.0
-4
-7
0
2
-4
-2
4
Figure 1: The empirical distributions of the weighted and unweighted least-squaree estimators for
(P) == Ep [1)] in two arm bandit setting where [E[YY, [Y [0)] O. We perform Thompson Sampling with (0, priors, (V 1) errors, and T = 1000. Specifically, we plot
E A,(5 (P)) the left and ( ET
Ar) (ÓAN' 0f (P)) on the right. (I)
0.5
The square-root importance weights are a form of variance stabilizing weights, akin to those introduced in Hadad et al. 2019 for estimating means and differences in means on data col- lected with multi-armed bandits. In fact, in the special case that |A] < do and s(X,, Ac) = [1A,=1, =2,=2, ... |4)= T, the adaptively weighted least-squares estimator equivalent to the weighted average estimator of Hadad et al. (2019. See Section g for more on Hadad et al. 2019
3.2 Asymptotic Normality and Confidence Regions
We now discuss conditions under which the adaptively weighted M-estimators are asymptotically normal in the sense of Equation 4h. In general, our conditions differ from those made for standard M-estimators on i..d. data because (i) the data is adaptively collected, i.c., M, can depend on H4-1 and (ii) we ensure uniform convergence over P P. which is stronger than guarantecing convergence pointwise for each P E P.
bidd Condition 1 (Stochastic Bandit Environment). Potential outcomes (X., Ya(a) a E A] ~ PEP over E T.
Condition I Qimplies that Y, is independent of H4-1 given X, and A,, and the conditional distribution Y, Xx. A, is invariant over time. Also note that action space can be finite or infinite.
Condition 2 (Differentiable). The first three derivatives of me(y. V, a) with respect to 0 exist for every 0 E θ, every E A. and every (I, y) in the joint support of (P: P.P. P.P...
Condition 3 (Bounded Parameter Space). For all P P, (P) E θ, bounded open subset of Rd. Condition 4 (Lipschitz). There exists some real-valued function g. such that (i)
suPpeP.121 1p. X,, A.)²] is bounded and (ii) for all 6, 0' e
(Y,,Y,, At) mo mo.(Y,, Xz, Ad) S XY,, A:) 0'1
5

---

Conditions E and 9 together restrict the complexity of the function m in order to ensure martingale law of large numbers result holds uniformly over functions {ma : E θ) this is used to prove the consistency of 0y. Similar conditions are commonly used to prove consistency of M-estimators based on i.i.d. data, although the boundedness of the parameter space can be dropped when me is concave function for all Y, Ar X2 (as it is in many canonical examples such as least squares) [Van der Vaart 2000 Engle 1994. Bura al. 2018]; we expect that a similar result would hold for adaptively
weighted M-estimators.
Condition 5 (Moments). The fourth moments of ma-(p)) (Y,, XY, At), ing- (Y,, Xt, At), and As) with respect to P and policy Ti are bounded uniformly over P E P and t 2 1. For all sufficiently large T, the minimum eigenvalue of Er,p 1 T Ei= Ep, (X,,, (0) is bounded above ónz > for all P P
mgg. (P) (Y,, X1,
Condition 5 is similar to those of Van der Vaart (000) Theorem 5.41]. However, to guarantee uniform convergence we assume that moment bounds hold uniformly over P P and f. 12 1.
Condition 6 (Third Derivative Domination), For B E Rdxdxd, Bdxxxdd such that (i)
define |BB||
EL Ej= [B..Ahl. There exists function Xi,Y,XX
supper,121 Ep [Y Xt, [b) is bounded and (ii for all P E tthere exists some ema >> 0 such that the following holds with probability 1,
sup 1 8 (P)l
XY, X1, A,)l S M A,)
Condition 6 is again similar to those in classical M-estimator asymptotic normality proofs [Van der Vaart 2000 Theorem 5.41].
Condition 7 (Maximizing Solution).
(i) For all P E P, there exists a (P) E such that (a) (P) E argmaxsce Ep [M, X1, ( Xr, Ar] w.p. 1, (b) Ep (P) Xx, X Xx, Ar] = 0 w.p.
and (c) Ep (p) (Y)) (Y1, At) Xt.
Ad| = w.p. 1.
(ii) There exists some positive definite matrix H such that + ET. E [ (P) Xr, A.]] =
H for all P E P and all sufficiently large T.
For matrices A, B, we define A 2 B to mean that B is positive semi-definite, as used above. Condition 7 (i) ensures that (P) is conditionally maximizing solution for all contexts X, and actions AG; this ensures that (ma- (P) (Y,, Ax) is a martingale difference sequence with respect to {H, Note it does not require (*(P) to always be a conditionally unique optimal solution. Condition 7 (ii) is related to the local curvature at the maximizing solution and the analogous condition in the i..d. setting is trivially satisfied; we specifically use this condition to ensure we can replace M(O((P) with (rrr in our asymptotic normality result, i.e., that (((0()))) 2s la uniformly over P.P.
Condition 8 (Well-Separated Solution). For all sufficiently large T, for any e >> O, there exists some d > 0 such that for all P
1 T
inf
Ep [Mo-(p)(Y,, At) mo - X1, Ai] 20. 6.
A well-separated solution condition akin to Condition g is commonly assumed in order to prove consistency of M-estimators, e.g., see Van der Vaart 2000 Theorem 5.7]. Note that the difference between Condition 7 (i) and Condition is that the former
conditional statement (conditional on X,, At) and the latter is a marginal statement (marginal over Xe, Ac, where A, is chosen according to stabilizing policies n² Condition 7 (i) means there is a 6" (P) solution for all contexts X, and
actions A, that does not need to be unique, however Condition g assumes that marginally over X,, A, there is well-separated solution.
(Tu1 Condition 9 (Bounded Importance Ratios). { Jt=1 do not depend on data (Ye, Xx, Ar) For x5 (i) & Pinax w.p. for some constants 0 < Puin S Pmax < 00.
all t 2 1, Pmin

---

Note that Condition 9 implies that for a stabilizing policy that is not tiryevarying, the action selection must be bounded away from zero w.p.
probabilities of the bandit algorithm IT, (,AA,,X; H,-1)
Similar boundedness assumptions are also made in the off-policy evaluation literature (Thomas and
Brunskill 20T6 Kallus and Uehara 2020 We discuss this condition further in Sections 3.3 and d
Theorem (Uniform Asymptotic Normality of Adaptively Weighted M- M-Estimators). Under Condi- we have that Ör E 0" (P) uniformly over P P. Additionally,
tions
- Er(P) 11/2)-12200 1/2 (0r) -P)) 2 N (0, Ia) uniformly over P E P.
(8)
The asymptotic normality result of equation
guarantees that for d-dimensional 0 (P).
lim inf inf Pp," T-+00 PEP
( (22
(P))
xxa xa(1-a)) = =1 -
Above x²,(1-)) is the 1 a quantile of the x² distribution with d degrees of freedom. Note that the region Cr {0 E e (P) (T0 0 xā,1--a) defines
d-dimensional syper-ellipsoid confidence region for 8" (P). Also note that since (0r) does not concentrate under standard bandit algorithms, we cannot use standard arguments to justify treating 0r as multivariate normal with covariance - (P) (rrr) - Nevertheless, Theorem can be used to guarantee valid confidence regions for subset of entries A (P) by using projected confidence regions [Nickerson] [994]. Projected confidence regions take confidence region for all parameters 0* (P) and project it onto the lower dimensional space on which the subset of target parameters lie (Appendix A.2
3.3 Choice of Stabilizing Policy
When the action space is bounded, using weights W; 11 H4-)) is equivalent to using square-root importance weights with stabilizing policy that selects actions uniformly over AA; this is because weighted M-estimators are invariant to all weights being scaled by the same constant. It can make sense to choose non-uniform stabilizing policy order to prevent the square-root importance weights from growing too large and to ensure Condition holds; disproportionately up-weighting few observations can lead unstable estimators. Note that an analogue of our stabilizing policy exists in the causal inference literature, namely, "stabilized weights" use probability density in the numerator of the weights to prevent them from becoming too large (Robins et al., 2000).
We now discuss how to choose stabilizing policies (m²** 122 in order to minimize the asymptotic variance of adaptively weighted M-estimators. We focus on the adaptively weighted least-squares estimator when we have a linear outcome model (Y]]), A.] = X, AA.:
T W. -Y- TA.) { 1=1
( Σ T
BAW-LS = argmax #ee
Recall that our use of adaptive weights is to adjust for instability in the variance of M-estimators induced by the bandit algorithm in order construct valid confidence regions; note that weighted estimators are not typically used for this reason. On i.i.d. data, the veast-squares criterion is weighted like in Equation ( in order to minimize the variance of estimators under noise heteroskedasticity; in this setting, the best linear unbiased estimator has weights W,
where o² (A,, X,) [p((Y, - x, 0%, (P))² X,, Ar]; this up-weights the importance of observations with low noise variance. Intuitively, if we do not need to variance stabilize, {We) should be determined by the relative importance of minimizing the errors for different observations, i.e., their noise variance.
In light of this observation, we expect that under homoskedastic noise there no reason to up-weight some observations over others. This would recommend choosing the stabilizing policy to make W.
/ (A,, X.)/ Ae, X,, 11 as close to 1 as possible, subject to the constraint that the stabilizing policies are pre-specified, i.c., { J:21 do not depend on data Xt, (see Appendix C for details). Since adjusting for heteroskedasticity and variance stabilization are distinct uses of weights, under heteroskedasticity, we recommend that the weights are combined in the following
sense: W.- = (1/o² X,)) (A, X,)/ \ (A,, X,, H4-1). This would mean that to minimize variance, we still want to choose the stabilizing policies to make n (Ar, X,)/ (A,, X,, H--1) as close to 1 possible, subject to the pre-specified constraint.
7

---

4 Related Work
Villar et al. 2015 and Raflerty et al. 2019 empirically illustrate that classical ordinary least squares (OLS) inference methods have inflated Type-1 error when used on data collected with variety of regret-minimizing multi-armed bandit algorithms. [hen et al. 2020 prove that the OLS estimator is asymptotically normal data collected with an e-greedy algorithm, but their results do not cover settings in which there is no unique optimal policy, e.g., a multi-arm bandit with two identical arms (Appendix E). Recent work has discussed the non-normality of OLS on data collected with bandit algorithms when there is no unique optimal policy and proposed alternative methods for statistical inference. A common thread between these methods is that they all utilize form of adaptive weighting. Deshpande et al. [2018] introduced the W-decorrelated estimator, which adjusts the OLS estimator with sum of adaptively weighted residuals. In the multi-armed bandit setting, the W-decorrelated estimator up-weights observations from early in the study and down-weights observations from later in the study (Zhan et al. 20201. In the batched bandit setting, Zhang et al. [2020] show that the Z-statistics for the OLS estimators computed separately on each batch are jointly asymptotically normal. Standardizing the OLS statistic for each batch effectively adaptively re-weights the observations in each batch.
Hadad ct al. 2 introduce adaptively weighted versions of both the standard augmented -ivversse propensity weighted estimator (AW-AIPW) and the sample mean (AWA) for estimating parameters of simple models on data collected with bandit algorithms. They introduce class of adaptive "variance stabilizing" weights, for which the variance of a normalized version of their estimators converges in probability to a constant. In their discussion section they note open questions, two of which this work addresses: 1) "What additional estimators can be used for normal inference with adaptively collected data?" and 2) How do their results generalize more complex sampling designs, like data collected with contextual bandit algorithms? We demonstrate that variance stabilizing adaptive weights can be used to modify a large class of M-estimators to guarantee valid inference. This generalization allows us to perform valid inference for a large class of important inferential targets: parameters of models for expected outcomes that are context dependent.
Recently, adaptive weighting has also been used in off-policy evaluation methods for when the behavior policy (policy used to collect the data) is contextual bandit algorithm [Bibaut et al. 2021 Zhan et al., 2021]. In this literature the estimand is the value, or average expected reward, of a pre- specified policy (note this is scalar value). In contrast, in our work we are interested in constructing confidence regions for parameters of a model for an outcome (that could be the reward)-for example, this could be parameters of logistic regression model for binary outcome. We believe in the future there could be theory that could unify these adaptive weighting methods for these different estimands.
An alternative to using asymptotic approximations to construct confidence intervals is to use high- probability confidence bounds. These bounds provide stronger guarantees than those based on asymptotic approximations, as they are guaranteed to hold for finite samples. The downside is that these bounds are typically much wider, which is why much of classical statistics uses asymptotic approximations. Here we do the same. In Section 5 we empirically compare our the self-
normalized martingale bound (Abbasi-Yadee el al. 201 high-probability bound commonly used in the bandit literature.
Simulation Results
In this section, R₁- = Yy. We consider two settings: continuous reward setting and a binary reward setting. In the continuous reward setting, the rewards are generated with mean [p[R,] Ar] = (P) + A,X, A (P) and noise drawn from a student's t distribution with five degrees of freedom; here X, [1, X.] E R3 (X, with intercept term), actions A, {0, 1}. and parameters (P)) E R33 In the binary reward setting, the reward RG is generated as a Bernoulli with -x,TT
success probability Ep R, X,, A,] = [1 + exp(-X: 1ep(-XXTB0 (P) both simulation settings optimal arm; we call vector parameter 0 (P) the advantage of selecting A, = 1 over A, = 0. Also in both settings, the contexts X, are drawn i.i.d. from a uniform distribution.
- A,X, 81 (P))]_1 Furthermore, in set 6% (P) [0.1. 0.1, 0.1] and (P) = |0, 0, 0), there is no unique
In both simulation settings we collect data using Thompson Sampling with linear model for the expected reward and normal priors [Agrawal and Goyal 2013 (so even when the reward is
8

---

binary). We constrain the action selection probabilities with clipping at rate of 0.05; this means that while typical Thompson Sampling produces action selection probabilities मt (Ar, X,, H2-1), we instead use action selection probabilities Ti (Ac, X,, H4-1) = 0.05 V (0.95 mे 4 (Ac, X1, H1-1)) to select actions. We constrain the action selection probabilities in order to ensure weights W, are bounded when using uniform stabilizing policy; see Sections 3.2] and p for more discussion on this boundedness assumption. Also note that increasing the amount the algorithm explores (clipping) decreases the expected width of confidence intervals constructed on the resulting data (see Section O
To analyze the data, in the continuous reward setting, we use least-squares estimators with a correctly specified model for the expected reward, i.e., M-estimators with mo RR, Xt, AA) = - -(R, - X, 0o A,X,T0;) We consider both the unweighted and adaptively weighted versions. We also compare to the seff-normalized martingale bound [Abbasi-
al. 2 and the W-decorrelated estimator [Deshpande et al. 2018). as they were both developed for the linear expected reward setting. For the self-normalized martingale bound. which requires explicit bounds on the parameter space, we set θ = (0 E I² ||01|2 < 6}. In the binary reward setting, we also assume correctly specified model for the expected reward. We use both unweighted and adaptively weighted maximum likelihood estimators (MLEs), which correspond to an M-estimators with mo Re, X A,) set to the negative log- rigllikelihood of R₂ given X,, A, We solve for these estimators using Newton-Raphsee optimization and do not put explicit bounds on the parameter space ② (note in this case mg is concave in 0 Aggrsti 2015 Chapter 5.4.2)). See Appendix A for additional details and simulation results.
In Figure we plot the empirical coverage probabilities and volumes of 90% confidence regions for 0* (P) (P)) 07 (P)) and (f(P) in both the continuous and binary reward settings. While the confidence regions based on the unweighted least-squares estimator (OLS) and the unweighted MLE have significant undercoverage that does not improve as T increases, the confidence regions based on the adaptively weighted versions, AW-LS and AW-MLE, have very reliable coverage. For the confidence regions for 01 (P) based on the AW-LS and AW-MLE, we include both projected confidence regions (for which we have theoretical guarantees) and non-projected confidence regions. The confidence regions based on projections are conservative but nevertheless have comparable volume to those based on OLS and MLE respectively. We do not prove theoretical guarantees for the non-projection confidence regions for AW-LS and AW-MLE, however they perform well across in our simulations. Both types of confidence regions based on AW-LS have significantly smaller volumes than those constructed using the self-normalized martingale bound and W-decorrelated estimator. Note that the W-decorrelated estimator and self- noflanordl martingale bounds are designed for linear contextual bandits and are thus not applicable for the logistic regression model setting. The confidence regions constructed using the self-normalized martingale bound have reliable coverage as well, but are very conservative. Empirically, we found that the coverage probabilities of the confidence regions based on the W-decorrelated estimator were very sensitive to the choice of tuning parameters. We use 5, 000 Monte-Carlo repetitions and the error bars plotted are standard errors.
6 Discussion
Immediate questions We assume that ratios n. (A,, ( (A,, X,, H2-1) are bounded for our theoretical results; this precludes Ty (A,, X,, He-1) from going to zero for a fixed stabilizing policy. For simple models, e.g., the AW-LS estimator, we can let these ratios grow at certain rate and still guarantee asymptotic normality (Appendix
B.5) we conjecture similar results hold more generally.
Generality and robustness This work assumes that have well- well-spedified model for the out- come Y,, i.e., that 8* (P) E argmax [Y, X,, Ax) A,) A,] w.p. 1. Our theorems use this assumption to ensure that { (Wiin (Y,, Xx, A.) is martingale difference sequence with respect to (He) On i.i.d. data it is common to define 0* (')P) to be the best projected solution, i.e., (P) argmaxpce Ep,, (moo.. Xt, A.)]. Note that the best projected solution, 0" (P), depends on the distribution of the action selection policy π. It would be ideal to also be able to perform inference for projected solution on adaptively collected data.
Another natural question is whether adaptive weighting methods work in Markov Decision Pro- cesses (MDP) environments. Taking the AW-LS estimator introduced in Section 3.1 as an example, our conditional variance derivation in Equation 7 fails to hold in an MDP setting, specifically equality (c). However, the conditional variance condition can be satisfied if we instead use weights (X.))) (X.)l/(m, (A,, X1, HP-1)P (X, X X,1,,11101 1)-1)) 1/2 where IP are the state
W. = (l (Ae, Xp)

---

90% Confidence Regions
Linear Model All Parameters
Linear Model Advantage Parameters
Logistic Model All Parametens
Logistic Model Advantago Parameters
0.95-
OLS MLE
-WW-LS /WW-LE
0.85
0.80-
AW-LS /AW-MLE
(Projected)
10g
WW-eecorrelaeed
(apese Bei) ewnoo
0.1g
Sell-Normalized
Martingala Bound
0.001
0
250
800
750
1000
250
500
750 1000 0 250
500
750
11000
250
500
750
1000
limesteps
Figure 2: Empirical coverage probabilities (upper row) and volume (lower row) of 90% confidence ellipsoids. The left two columns are for the linear reward model setting (1-distributed rewards) and the right two columns are for the logistic regression model setting (Bernoulli rewards). We consider confidence ellipsoids for all parameters 0* (P) and for advantage parameters 0f (P) for both settings.
transition probabilities and pS is pre-specified distribution over states. In general though we do not expect to know the transition probabilities Pp and if we tried to estimate them, our theory would require the estimator to have error (1/1/v) /T), below the parametric rate.
Trading-off regret minimization and statistical inference objectives In sequential decision- making problems there is a fundamental trade-off between minimizing regret and minimizing estimation error for parameters of the environment using the resulting data [Bubeck al. 2009 Dean et al. 2018). Given this trade-off there are many open problems regarding how to minimize regret while still guaranteeing a certain amount of power or expected confidence interval width, e.g.. developing sample size calculators for use in justifying the number of users in mobile health trial, and developing new adaptive algorithms [Lu al. 2014
Erraqabi al. 2017
Yao et 2020
Acknowledgements and Disclosure of Funding
We thank Yash Nair for feedback early drafts of this work.
Research reported in this paper was supported by National Institute on Alcohol Abuse and Al- coholism (NIAAA) of the National Institutes of Health under award number R013A2218,, Na- tional Institute on Drug Abuse (NIDA) of the National Institutes of Health under award number P50DA039838, National Cancer Institute (NCI) of the National Institutes of Health under award number U01CA229437, and by NIH/NIBIB and OD award number P41EB028242. The content solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGEI745303. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
Yasin Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.
Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135, 2013.
Alan Agresti. Foundations of linear and generalized linear models. John Wiley & Sons, 2015.
10

---

Aurélien Bibaut, Antoine Chambaz, Maria Dimakopoulou, Nathan Kallus, and Mark van der Laan. Post-contextual-bante inference. NeurIPS 2021, 2021.
Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In International conference Algorithmic learning theory, pages 23-37. Springer, 2009.
Efstathia Bura, Sabrina Duarte, Liliana Forzani, Ezequiel Smucler, and Mariela Sued. Asymptotic theory for maximum likelihood estimates in reduced-rank multivariate generalized linear models. Statistics, 52(5): 1005-440 2018.
Stefano Caria, Maximilian Kasy, Simon Quinn, Soha Shami, Alex Teytelboym, et al. An adaptive targeted field experiment: Job search assistance for refugees in jordan. 2020.
Haoyu Chen, Wenbin Lu, and Rui Song. Statistical inference for online decision making; a contextual bandit setting. Journal of the American Statistical Association, pages 1-16, 2020.
Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. In Advances in Neural Information Processing Systems, 2018.
Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive linear models. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1194-1203, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Aryeh Dvoretzky. Asymptotic normality for sums of dependent random variables. In Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume Probability Theory. The Regents of the University of California, 1972.
Robert F Engle. Handbook of econometrics: volume 4. Number 330.015195 E53 v. 4. 1994.
Akram Erraqabi, Alessandro Lazarie, Michal Valko, Emma Brunskill, and Yun-En Liu. Trading off rewards and errors in multi-armed bandits. In Artificial Intelligence and Statistics, pages 709-717. PMLR, 2017.
Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. arXiv preprint 1rXivv1911.02211 0.0066, 2019.
John Hammersley. Monte carlo methods. Springer Science & Business Media, 2013.
Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy eval- uation in markov decision processes. Journal of Machine Learning Research, 21(167):1-63, 2020.
Maximilian Kasy. Uniformity and the delta method. Journal of Econometric Methods, 8(1), 2019.
Maximilian Kasy and Anja Sautmann. Adaptive treatment assignment in experiments for policy choice. Econometrica, 19(1):113-132, 2021.
Tze Leung Lai and Ching Zong Wei. Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics, 10(1): 164-166, 1982.
Hannes Leeb and Benedikt M Pötscher. Model selection and inference: Facts and fiction. Econometric Theory, pages 21-59, 2005.
Peng Liao, Kristjan Greenewald, Predrag Klasnja, and Susan Murphy. Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-22, 2020.
11

---

Yun-En Liu, Travis Mandel, Emma Brunskill, and Zoran Popovic. Trading off scientific knowledge and user learning with multi-armed bandits. EDM, pages 161-168, 2014.
David Nickerson. Construction of a conservative confidence region from projections of an exact confidence region in multiple linear regression. The American Statistician, 48(2):120-113 1994.
Anna Rafferty, Huiji Ying. and Joseph Williams. Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. JEDMI Journal of Educational Data Mining, 11(1):47-79, 2019.
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology, 2000.
Joseph P Romano, Azeem M Shaikh, et al. On the uniform asymptotic validity of subsampling and the bootstrap. The Annals of Statistics, 40(6):2798 2822, 2012.
Hammad Shaikh, Arghavan Modiri, Joseph Jay Williams, and Anna Rafferty. Balancing student success and inferring personalized effects in dynamic experiments. In EDM, 2019.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.
Aad W Van der Vaart. Asymptotic Statistics, volume 3. Cambridge University Press, 2000.
Aad W Van Der Vaart and Jon A Wellner. Weak convergence. In Weak convergence and empirical processes, pages 16-28. Springer, 1996.
Soffa S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical science: review journal of the Institute of Mathematical Statistics, 30(2):199, 2015.
Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and adaptive off- off-policy evaluation in contextual bandits. In International Conference Machine Learning, pages 3589-3599 PMLR, 2017.
Jiayu Yao, Emma Brunskill, Weiwei Pan, Susan Murphy, and Finale Doshi Velez. Power-constrained bandits. arXiv preprint M.XX::20004062330 2020.
Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encouraging physical activity in patients with diabetes: intervention using reinforcement learning system. Journal of medical Internet research, 19310);333, 2017.
Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021.
Kelly W Zhang, Lucas Janson, and Susan A Murphy. Inference for batched bandits. In Advances in Neural Information Processing Systems, 2020.
12


Reviews:
[{"review_id": "mD0V_APq0s", "reviewer": "Reviewer_55GN", "summary": "The authors analyze square root importance sampling weighted M-estimators over parametric classes for adaptively collected data. \n\nThis work is motivated by the fact that the usual unweigthed M-estimators that one would use for i.i.d. data fail to be asymptotically normally distributed under adaptively collected data.\n\nUnder appropriate conditions, the proposed weighting scheme makes the resulting M-estimator asymptotically normal, uniformly over the parameter space.\n\nThe authors demonstrate their claims through simulations.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is clearly written, even though I see some directions of improvement which I list further down.\n- Inference from adaptively collected data is well motivated and of great practical significance.\n- The contribution is novel to the best of my knowledge.\n- Most conditions are well discussed.\n- I've checked the proofs and they are correct.\n- Simulations are compelling. In particular the existing baseline methods are clearly inferior and non-usable for inference.\n\n\n\n### Significance:\n\nThe main weakness is that the guarantees do not seem to hold under model misspecification, which reduces the scope and practical applicability the contribution. I discuss this further in the next text box. I will update my rating depending on answers to this concern. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems to me that condition 7 easily breaks when the reward model is incorrectly specified. \n\n(Say for instance that $m_\\theta(x,a,y) = -(y - \\theta^\\top_a x)^2$. Suppose that $ X_1^{(1)},...,X_d^{(1)} $ and $ X_1^{(2)},...,X_d^{(2)} $ are two subsequences of the context sequence corresponding to rounds with $A_t = 1$, such that the corresponding empirical covariance matrices are full-rank, Then, if condition 7 holds, for $a=1$, $\\theta^*_{a, \\mathcal{P}}$ will be a minimizer of both \n\n$$ \\sum_{s=1}^d E[(Y^{(1)}_s - \\theta^\\top_a X^{(1)}_s | X^{(1)}_s]$$ \n\nand \n\n$$ \\sum_{s=1}^d E[(Y^{(2)}_s - \\theta^\\top_a X^{(2)}_s | X^{(2)}_s].$$\n\nSince the design matrices $(X_1^{(1)},...,X_d^{(1)})^\\top $ and $(X_1^{(2)},...,X_d^{(1)})^\\top $ \nare full rank, the minimizers are unique, and are a priori different if the model is misspecified and the design matrices are distinct.)\n\nI think the authors should clearly state that in general condition 7 does not hold under misspecification. I think this is the main caveat of this work which has serious practical implications, as parametric models are hardly ever correctly specified in practical applications.\n\nThis also raises another issue. Even if we had guarantees for inference on the coefficients of a misspecified parametric model, I think the practical usefulness of such guarantees of such results is limited. For instance this cannot be used to perform causal inference on the mean potential outcome of a given an arm, as no function of the coefficients equals that statistical parameter in general if the model is misspecified.\n\n\n## Additional minor comments:\n\nLine 297: \nWhy do these bounds need to be anytime ?  The asymptotic normality result stated in the article allows to construct an approximate CI at a given sample size, but if you wanted to give an approximate anytime confidence sequence, you’d need a stronger type of asymptotic distributional result (such as a weak invariance principle) that would characterize the joint asymptotic distribution of the sequence of estimators. \nIn that sense, it seems to me that the finite sample counterpart of your work is a concentration bound at a specific time point - not anytime. \n\nLine 143-144 and 149-154. I think these identities would be better stated as lemmas and proven in appendix. \n\nLine 187: There is an issue with the sentence.\n\nLine 195: $\\dot{m}$ and $\\ddot{m}$ are not defined (although their meaning is clear)\n\nLine 201: Condition 6: I think it'd be nice to define $\\left\\lVert B \\right\\rVert_1$ at the beginning of the assumption statement, not at the end, so that the reader doesn't run into this notation before it is defined.\n\nLine 305: Minor comment: I think there is no need to introduce the additional notation $R_t$. You could just say that the outcome $Y_t$ has the interpretation of a reward. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "lKXQ3-IUGj4", "reviewer": "Reviewer_GB2D", "summary": "This paper introduces an M-estimation-based method that provides valid inference for adaptively collected data. In particular, the contribution of the new inference technique includes: 1) it applies to a wide range of estimands that can be written as the maximizer of a loss function; 2) it provides valid inference in the presence of contexts; 3) uniform convergence is established.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is in general a well-written paper, and I find its contribution to the inference technique for adaptively collected data quite significant. I have two major questions regarding the paper:\n\n1. It is assumed in Condition 9 that the ratio between \\pi^{sta} and \\pi_t is bounded from above and below by a constant. If, as the authors suggest at the beginning of Section 3, we take \\pi^{sta} to be 1/|A|, this is essentially the overlap condition. This immediately assumes away many technical difficulties and does not seem to be very natural, since the main feature of the adaptively collected/bandit-collected data is that there are vanishing propensity scores. Is it possible to improve on this condition? If not, what are the difficulties?\n\n2. I notice that there is a parallel paper (Zhan et al., 2021) that studies a similar problem. Their paper allows the propensity score to go to zero, but the results they provide are different from this paper. I am curious to see if there is any equivalence and difference between the results given by the two papers.\n\nReference:\n\nZhan, Ruohan, et al. \"Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits.\" arXiv preprint arXiv:2106.02029 (2021).\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A discussion on the potential negative societal impact of the work should be added.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Q9sJULFk0T", "reviewer": "Reviewer_yw7d", "summary": "This paper extends classical M-estimator results to settings where the data are collected sequentially.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From a technical perspective, the contributions seem relatively unsurprising, combining classical M-estimator conditions with the notion of stabilizing weights arising in the bandit/optimal treatment literatures. That said, putting these technical arguments together correctly requires expertise that wouldn't be available to quite a few potential users of this approach. Also, the uniformity established by the authors is also useful for helping to ensure that the asymptotics will provide meaningful finite sample insights. So, I have an overall favorable view of this work. \n\nI have a few comments/questions:\n\n1. It would be worth describing cases where the condition that $\\theta^*(P)$ exists for a given $m_\\theta$ (below Eq. 1) is expected to hold, and also settings where it is not expected to hold. E.g., in a correctly specified parametric/semiparametric model is this always plausible provided that $\\theta^*(P)$ is some (function) of the indexing parameter? How about more generally?\n\n2. Along the same lines as above, could you develop your results for the case where the objective function for $\\theta^*(\\mathcal{P})$ is defined as some kind of average over possible values of $(X_t,A_t)$ or so, resulting in the estimand being some kind of least squares/KL projection that doesn't rely on the existence of a $\\theta^*(P)$ that satisfies the almost sure assumption in Eq. 1?\n\n3. The theoretical section allows the action space $\\mathcal{A}$ to be finite or infinite, but then the experiments all focus on the case were $\\mathcal{A}=\\{0,1\\}$. It would be helpful to see an example with a larger $\\mathcal{A}$, to provide some assurance that the asymptotics provided are actually relevant at realistic sample sizes (the current uniform theoretical results show that a sample size exists at which the asymptotic results are relevant, but doesn't tell you what that sample size actually is).\n\nClarifying questions/comments:\n\n1. On pg 5 line 192, it says \"when $m_\\theta$ is concave\". Concave in what? In $\\theta$ for a fixed $x,a,y$, or something else?\n\n2. I think that Condition 7 should read: \"For all $P\\in\\mathcal{P}$, there exists a $\\theta^*(\\mathcal{P})\\in\\Theta$ such that $\\theta^*(\\mathcal{P})\\in\\arg\\max_{\\theta\\in\\Theta}\\mathbb{E}_{\\mathcal{P}}[m_\\theta(Y_t,X_t,A_t)|X_t,A_t]$ w.p. 1.\"\n\nThere also appears to be a typo at the bottom of page 2: $f\\hat{\\theta}$.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "Jraj2oqYxXy", "reviewer": "Reviewer_H36N", "summary": "This paper studies how to conduct statistical inference with M-estimators based on data collected from adaptive algorithms. It proposes an adaptively weighted M-estimator with square-root importance weights and proves that under fairly standard assumptions, the estimator (uniformly) has asymptotic normal distribution and it can be used to construct uniformly valid confidence intervals. By studying M-estimation problems, this paper substantially broadens the scope of estimation problems in this recent literature of statistical inference with adaptively collected data. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies statistical inference with adaptively collected data, a very important problem that attracts a lot of recent attention. I think this paper's study of general M-estimation problems, not just only simple expectations or linear models, makes a significant contribution in this area. Moreover, this paper is well-written, and it is overall easy to follow. I particularly section 3.1 about the intuition for the proposed weights, which is really helpful, and section 6 that discusses many questions I had while I was reading sections before this part. \n\nI only have some minor comments:\n\n1. eq (7) doesn't look right: it seems that from equalities (a) on, there shouldn't be an indicator function (At = a), right? It should cancel with the inverse propensity weights in the denominator, and instead the inner conditional expectation should be conditioned on (At = a) instead. \n\n2. I'm a bit confused by condition 7 (i) and condition 8. The authors state that the parameter thata^* is not necessarily unique sol to the conditional maximization problem, but condition 8 assumes an identification condition, which usually needs uniqueness. How should I understand this? So does this mean that even though theta^* is nonunique, the stablized expectation in condition (8) somehow rules out all of them but only a single identifiable one? If this is the case, then I do not think it is very plausible, although I get that the current presentation is in some sense more general than assuming unique solution to the conditional maximization problem. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. They discuss the limitations in their section 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The expert reviewers all appreciated the paper and agree it provides useful results and that the paper should be accepted. The authors are to be congratulated for an interesting contribution that nicely handles a very timely topic. The authors are expected to address the points raised by reviewers in a final version, including as they outlined in their response. In my opinion the most important thing to address is to make very clear early on the setting where the method is appropriate, namely well-specified parametric models, and that this excludes, for example, a constant model for the mean reward of an arm or policy. This is not to detract from the work -- rather, making clear what the paper sets out to do and what it does not will significantly improve clarity for the reader and therefore the paper's impact. A discussion regarding what would actually happen under misspecification (and whether slight misspecification results in only slight aberrations from the results) would potentially add a lot too (I believe the target estimand would then depend on the logging policies, if I understand correctly).
Decision: Accept (Poster)