Title: Off-policy estimation with adaptively collected data: the power of online learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causal inference

function approximation

learning theory

off-policy estimation

online learning

reinforcement learning
Abstract: We consider estimation of a linear functional of the treatment effect from adaptively collected data. This problem finds a variety of applications including off-policy evaluation in contextual bandits, and estimation of the average treatment effect in causal inference. While a certain class of augmented inverse propensity weighting (AIPW) estimators enjoys desirable asymptotic properties including the semi-parametric efficiency, much less is known about their non-asymptotic theory with adaptively collected data. To fill in the gap, we first present generic upper bounds on the mean-squared error of the class of AIPW estimators that crucially depends on a sequentially weighted error between the treatment effect and its estimates. Motivated by this, we propose a general reduction scheme that allows one to produce a sequence of estimates for the treatment effect via online learning to minimize the sequentially weighted estimation error. To illustrate this, we provide three concrete instantiations in (1) the tabular case; (2) the case of linear function approximation; and (3) the case of general function approximation for the outcome model. We then provide a local minimax lower bound to show the instance-dependent optimality of the AIPW estimator using no-regret online learning algorithms.

Off-policy estimation with adaptively collected data: the power of online learning

Abstract

We consider estimation of a linear functional of the treatment effect from adaptively collected data. This problem finds a variety of applications including off-policy evaluation in contextual bandits, and estimation of the average treatment effect in causal inference. While a certain class of augmented inverse propensity weighting (AIPW) estimators enjoys desirable asymptotic properties including the semiparametric efficiency, much less is known about their non-asymptotic theory with adaptively collected data. To fill in the gap, we first present generic upper bounds on the mean-squared error of the class of AIPW estimators that crucially depends on a sequentially weighted error between the treatment effect and its estimates. Motivated by this, we propose a general reduction scheme that allows one to produce a sequence of estimates for the treatment effect via online learning to minimize the sequentially weighted estimation error. To illustrate this, we provide three concrete instantiations in (1) the tabular case; (2) the case of linear function approximation; and (3) the case of general function approximation for the outcome model. We then provide a local minimax lower bound to show the instance-dependent optimality of the AIPW estimator using no-regret online learning algorithms.

1 Introduction

Estimating a linear functional of the treatment effect is of great importance in both causal inference and reinforcement learning (RL). For instance, in causal inference, one is interested in estimating the average treatment effect (ATE) [20] or their weighted variants, and in the literature of bandits and RL, one is interested in estimating the expected reward of a target policy [38, 64, 41, 37]. Two main challenges arise when tackling this problem:

Off-policy estimation : Oftentimes, one needs to estimate the linear functional based on observational data collected from a behavior policy. This behavior policy may not match the desired distribution specified by the linear functional [42];

Adaptive data collection mechanism : It is increasingly common for observational data to be adaptively collected due to the use of online algorithms (e.g., via contextual bandit algorithms [60, 33, 2, 52, 34]) in experimental design [67].

In this paper, we deal with two challenges simultaneously by investigating the estimation of a linear functional of the treatment effect from observational data that are collected adaptively. When the observational data is collected non-adaptively, i.e., in an i.i.d. manner, there is an extensive line of work [51, 49, 10, 24, 1, 27, 43, 6, 3, 64, 41] investigating the asymptotic and non-asymptotic theory of various estimators. Most notably are the study [6] that establishes the asymptotic efficiency of a family of semi-parametric estimators, and a more recent study [42] that undertakes a finite-sample

analysis which uncovers the importance of a certain weighted ℓ 2 -norm when estimating the treatment effect. On the other hand, when it comes to adaptively collected data, most prior works [16, 67] focus on the asymptotic normality of the estimators, and do not discuss the finite-sample analysis of the estimators. In this paper, we aim to fill in this gap.

1.1 Main contributions

More specifically, we make the following three main contributions in this paper:

First, we present generic finite-sample upper bounds on the mean-squared error of the class of augmented inverse propensity weighting (AIPW) estimators that crucially depends on a sequentially weighted error between the treatment effect and its estimates. This sequentially weighted estimation error demonstrates a clear effect of history-dependent behavior policies;

Second, motivated by previous finding, we propose a general reduction scheme that allows one to form a sequence of estimates for the treatment effect via online learning to minimize the sequentially weighted estimation error. To demonstrate this, we provide three concrete instantiations in (1) the tabular case; (2) the case of linear function approximation; and (3) the case of general function approximation for the outcome model;

In the end, we provide a local minimax lower bound to showcase the instance-dependent optimality of the AIPW estimator using no-regret online learning algorithms in the large-sample regime.

1.2 Related works

Off-policy estimation with observational data Off-policy estimation in observational settings has been a central topic in statistics, operations research, causal inference, and RL. Here, we group a few prominent off-policy estimators into the following three categories: (i) Model-based estimator : often dubbed as the direct method (DM), whose key idea is to utilize observational data to learn a regression model that predicts outcomes for each state-action pair, and then average these model predictions [29, 10, 9, 39]. Due to model mis-specification, DM typically has a low variance but might lead to highly biased estimation results. (ii) Inverse propensity weighting (IPW): for the OPE task, IPW uses importance weighting to account for the distribution mismatch between the behavioral policy and the target policy [21, 55]. If the behavioral policy differs significantly from the target policy, then IPW can have an overly large variance (known as the low overlap issue) [23]. Typical remedies for this issue include propensity clipping [25, 57] or self-normalization [19, 58]. (iii) Hybrid estimator : some off-policy estimators (e.g., the doubly-robust (DR) estimator [10]) combine DM and IPW together to blend their complementary strengths [48, 10, 9, 59, 12, 56, 64]. A key asymptotic results in OPE is that the cross-fitted DR is √ n -consistent and asymptotically efficient (that is, it attains the lowest possible asymptotic variance), even for the case where nuisance parameters are estimated at rates slower than √ n -rates [6]. However, these methods still might be vulnerable to the low overlap issue especially for large or continuous action spaces. Thus, there has been a line of recent studies on OPE for large action spaces [13, 53, 44, 54] and OPE for continuous action space [28, 35, 63].

Off-policy estimation with adaptively collected data A recent strand of works studied asymptotic theory of adaptive variants of the IPW and DR estimators (e.g., asymptotic normality, semi-parametric efficiency, and confidence intervals) [31, 8, 7] for adaptively collected data. However, in adaptive experiments, overlap between the behavioral policies and the target policy can deteriorate since the experimenter shifts the behavioral policies in response to what he/she observes (known as the drifting overlap ) [67]. It may engender unacceptably large variances of the IPW and DR estimators. To address this large variance problem, there has been a recent strand of works investigating variance reduction strategies for the DR estimator based on shrinking importance weights toward one [4, 64, 57, 56], local stabilization [40, 69], and adaptive weighting [17, 67]. Recent studies on policy learning with adaptively collected data [68, 26] explored the adaptive weighting DR estimator for policy learning. In contrast with the majority of prior works on off-policy estimation with adaptively collected data that focus on asymptotic results, this paper aims at establishing non-asymptotic theory of the problem. While several researchers have been recently explored non-asymptotic results of the problem with an emphasis on uncertainty quantification [30, 65], we focus on analyses of estimation procedures of the off-policy value. As a majority of existing standard objects for uncertainty quantification, such as a confidence interval (CI), take a very static view of the world (e.g., it holds for a fixed sample size and

is not designed for interactive/adaptive data collection procedures), the aforementioned two papers [30, 65] instead study a more suitable statistical tool for such cases called a confidence sequence .

2 Problem formulation

We first formulate our problem using the language of contextual bandits: let X , A , and Y ⊆ R denote the context space , the action space , and the outcome space , respectively. Denote by O := X × A × Y the space of all possible context-action-outcome triples. In an adaptive experiment, one observes n samples { ( X i , A i , Y i ) ∈ O : i ∈ [ n ] } produced by the following data generating procedure [26, 68]: At each stage i ∈ [ n ] ,

(i) Acontext X i ∈ X is independently sampled from a fixed context distribution Ξ ∗ ( · ) ∈ ∆( X ) ;

(ii) There exists a behavioral policy Π ∗ i ( · , · ) : X × O i -1 → ∆( A ) that selects the i -th action as A i | X i , O i -1 ∼ Π ∗ i ( · | X i , O i -1 ) , where O i := ( X 1 , A 1 , Y 1 , · · · , X i , A i , Y i ) ∈ O i for i ∈ [ n ] . As Π ∗ i ( · | X i , O i -1 ) may depend on previous observations, { ( X i , A i , Y i ) : i ∈ [ n ] } are no longer i.i.d.;

(iii) Given a Markov kernel Γ ∗ ( · , · ) : X × A → ∆( Y ) , we assume that the outcome is generated according to Y i ∼ Γ ∗ ( · | X i , A i ) . Moreover, the conditional mean of the outcome Y i ∈ Y is specified as

where the function µ ∗ ( · , · ) : X × A → R is called the treatment effect (in causal inference) or the reward function (in bandit and RL literature). We note that the treatment effect µ ∗ is not revealed to the statistician. We also define the conditional variance function σ 2 ( · , · ) : X × A → [0 , + ∞ ] defined by σ 2 ( x, a ) := E [ { Y -µ ∗ ( X,A ) } 2 ∣ ∣ ∣ ( X,A ) = ( x, a ) ] , which is assumed to satisfy σ 2 ( x, a ) < + ∞ for every state-action pair ( x, a ) ∈ X × A .

At this moment, we assume the existence of σ -finite base measures λ X ( · ) , λ A ( · ) , and λ Y ( · ) over X , A , and Y , resp., such that Ξ ∗ ( · ) ≪ λ X ( · ) , Π ∗ i ( · | x, o i -1 ) ≪ λ A ( · ) for every ( x, o i -1 ) ∈ X × O i -1 and i ∈ [ n ] , and Γ ∗ ( · | x, a ) ≪ λ Y ( · ) for all state-action pairs ( x, a ) ∈ X × A . Here, the notation ≪ stands for the absolute continuity of measures. Our main goal is to estimate the off-policy value for any given target evaluation function g ( · , · ) : X × A → R defined as

where I ∗ := (Ξ ∗ , Γ ∗ ) ∈ I := ∆( X ) × ( X × A → ∆( Y )) defines our problem instance . Throughout the paper, we assume that the propensity scores { π ∗ i ( X i , O i -1 ; A i ) : i ∈ [ n ] } are revealed, where π ∗ i ( x, o i -1 ; · ) := dΠ ∗ i ( ·| x, o i -1 ) d λ A : A → R .

As we mentioned earlier in Section 1, the estimation problem of a linear functional of the treatment effect µ ∗ turns out to be useful in both causal inference and RL in the following sense:

Estimation of average treatment effects : We consider the binary action space A = { 0 , 1 } equipped with the counting measure. The average treatment effect (ATE) in our problem setting is defined as the linear functional

Once we take the evaluation function as g ( x, a ) = 2 a -1 , the ATE boils down to a particular case of the equation (1);

Off-policy evaluation (OPE) for contextual bandits : Assume that a target policy Π target ( · ) : X → ∆( A ) is given such that Π target ( · | x ) ≪ λ A ( · ) for every context x ∈ X . For simplicity, let π target ( x, · ) := dΠ target ( ·| x ) d λ A denote the density function of the target policy for each context x ∈ X . If we take g ( x, a ) = π target ( x, a ) , then the linear functional (1) corresponds to the value of the target policy Π target . This problem has been widely studied in the literature of bandits and RL, known as the off-policy evaluation (OPE).

Weconclude this section by introducing notations that will be useful in later sections: let P i I ∈ ∆ ( O i ) denote the law of the sample trajectory O i under the sampling mechanism with a problem instance I = (Ξ , Γ) ∈ I . We denote the density function of P i I ∈ ∆ ( O i ) with respect to the base measure ( λ X ⊗ λ A ⊗ λ Y ) ⊗ i by p i I ( · ) : O i → R + . Lastly, we define the k -th weighted ℓ 2 -norm for k ∈ [ n ] as

for any function φ ( · , · ) : X × A → R , together with the k -th weighted ℓ 2 -space by

3 A class of AIPW estimators and non-asymptotic guarantees

The main objective of this section is to develop a meta-algorithm to tackle the estimation problem of the off-policy value (1), followed by some key rationale of the proposed procedure as a variancereduction scheme of the standard inverse propensity weighting (IPW) estimator.

3.1 How can we reduce the variance of the IPW estimator?

Akin to [42], we consider a class of two-stage estimators obtained from simple perturbations of the IPW estimator. Given any collection f := ( f i : X × O i -1 × A → R : i ∈ [ n ] ) of auxiliary functions, we consider the following perturbed IPW estimator ˆ τ f n ( · ) : O n → R :

For each i ∈ [ n ] , let ν i ∈ ∆ ( X × O i -1 × A ) denote the joint distribution of ( X i , O i -1 , A i ) induced by the adaptive data collection procedure described in Section 2. Then, we arrive at the following result whose proof is deferred to Appendix B.1:

Proposition 3.1. For any collection f := ( f i ∈ L 2 ( ν i ) : i ∈ [ n ] ) of auxiliary deterministic functions, we have E I ∗ [ ˆ τ f n ( O n ) ] = τ ( I ∗ ) . Furthermore, if

for each i ∈ [ n ] , then

From the decomposition (4) of the variance of the perturbed IPW estimate ˆ τ f n ( O n ) , one observes that the only term that depends on the collection of auxiliary functions f is the third term. More importantly, the third term is equal to zero if and only if

The collection of minimizing functions f ∗ := ( f ∗ i ∈ L 2 ( ν i ) : i ∈ [ n ] ) yields the oracle estimator ˆ τ f ∗ n ( · ) : O n → R

whose variance is given by

Algorithm 1 Meta-algorithm: augmented inverse propensity weighting (AIPW) estimator.

Require: the dataset D = { ( X i , A i , Y i ) ∈ O : i ∈ [ n ] } and an evaluation function g : X × A → R .

1: For each step i ∈ [ n ] , we compute an estimate ˆ µ i ( O i -1 ) ∈ ( X × A → R ) of the treatment effect based on the sample trajectory O i -1 up to the ( i -1) -th step. // Implement Algorithm 2 as a subroutine;

2: Consider the AIPW estimator (a.k.a., the doubly-robust (DR) estimator) ˆ τ AIPW n ( · ) : O n → R :

where the objects being averaged are the AIPW scores ˆ Γ i ( · ) : O i → R is defined by

3: return the AIPW estimate ˆ τ AIPW n ( O n ) .

3.2 The class of augmented IPW estimators

Since the treatment effect µ ∗ is not revealed to the statistician in (6), it is impossible to exactly compute the oracle estimate ˆ τ f ∗ n ( · ) : O n → R using only the observational dataset O n . Therefore, a natural remedy would be the following two-stage procedure, which is referred to as the augmented inverse propensity weighting (AIPW) estimator or the doubly-robust (DR) estimator [10, 50, 61, 17, 67, 22]: (i) we first compute a sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect µ ∗ ; and then (ii) we plug-in these estimates to the equation (6) to construct an approximation to the ideal estimate ˆ τ f ∗ n ( O n ) . We summarize this two-stage procedure in Algorithm 1.

We pause here to compare our problem setting and algorithms with the most relevant work [42]. We focus on off-policy estimation with adaptively collected data, which is technically more challenging compared to i.i.d. data considered in [42]. In the case with i.i.d. data, [42] proposed a natural approach to construct a class of two-stage estimators as follows: (a) compute an estimate ˆ µ of the treatment effect µ ∗ utilizing part of the dataset; and (b) substitute this estimate in the equation (6) of the oracle estimator. Note that the authors use the cross-fitting approach [5, 6], which allows to make full use of data to maintain efficiency and statistical power of machine learning algorithms for estimation of nuisance parameters while reducing overfitting bias. However, the cross-fitting strategy heavily relies on the i.i.d. nature of the data collection mechanism and therefore one cannot use it in the setting with adaptively collected data. Instead, we construct an estimate ˆ µ i of the treatment effect µ ∗ based on the sample trajectory O i -1 at each stage and then substitute these estimates in the equation (6). This is one of main contributions to address the adaptive nature of our data generating mechanism. We will make use of the framework of online learning to construct a sequence of estimates for the treatment effect µ ∗ .

3.3 Theoretical guarantees of Algorithm 1

In this section, we provide statistical guarantees for the class of AIPW estimators for dealing with the estimation problem of the off-policy value (1). The main result of this section can be summarized as the following non-asymptotic upper bound on the mean-squared error (MSE) of Algorithm 1:

Theorem 3.1 (Non-asymptotic upper bound on the MSE of the AIPW estimator) . For any sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } for the treatment effect µ ∗ , the AIPW estimator (8) has the MSE bounded above by

Note that the non-asymptotic upper bound (10) on the MSE for the class of AIPW estimators (8) consists of two terms, both of which have natural interpretations. The first term v 2 ∗ corresponds to the

Algorithm 2 Online non-parametric regression protocol for estimation of the treatment effect.

Require: the number of rounds n ∈ N

.

1: for i = 1 , 2 , · · · , n , do

2: The learner selects a point ˆ µ i ( O i -1 ) ∈ ( X × A → R ) based on the sample trajectory O i -1 ;

3: The environment then picks a loss function l i ( · ) : ( X × A → R ) → R defined as

4: end for

5: return the sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect.

optimal variance (7) achievable by the oracle estimator, and the second term

measures the average estimation error of the estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of µ ∗ . Of primary interest to us is a subsequent upper bounding argument based on the MSE bound (10) in the finite sample regime: in particular, to minimize the RHS of (10), one needs to choose a sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } which minimizes the second term (11).

3.4 Reduction to online non-parametric regression

Let us now focus on constructing a sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect and upper bounding the estimation error (11) in the MSE bound (10). To this end, we borrow ideas from the literature of online non-parametric regression [45].

To begin with, we consider an n -round turn-based game between the learner and the environment; see Algorithm 2 for the details. Then, one can readily observe for any µ ( · , · ) : X × A → R , we have

In the current turn-based game, our natural goal is to minimize the learner's static regret against the best fixed action in hindsight belonging to a pre-specified function class F ⊆ ( X × A → R ) :

where A denotes the learner's online non-parametric regression algorithm that returns a sequence of estimates { ˆ µ i ( O i -1 ) : i ∈ [ n ] } for the treatment effect. Then, one can establish the following oracle inequality that demystifies a relationship between estimation problem of the off-policy value and the online non-parametric regression protocol. See Appendix B.3 for the proof.

Theorem 3.2 (Oracle inequality for the class of AIPW estimators) . The AIPW estimator (8) using the sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect µ ∗ produced by the online non-parametric regression algorithm A enjoys the following upper bound on the MSE:

A few remarks are in order. Apart from the optimal variance v 2 ∗ , the RHS of the bound (15) contains two additional terms: (i) the expected regret relative to the number of rounds n , where the expected value is taken over O n ∼ P n I ∗ ( · ) ; and (ii) the approximation error under the ∥·∥ ( n ) -norm. Given any fixed function class F ⊆ ( X × A → R ) , if we consider the large sample size regime, i.e., the sample

size n is sufficiently large, then one can see that the asymptotic variance of the AIPW estimator (8) is asymptotically the same as v 2 ∗ +inf { ∥ µ -µ ∗ ∥ 2 ( n ) : µ ∈ F } , provided that the online non-parametric regression algorithm A exhibits a no-regret learning dynamics , i.e., E I ∗ [ Regret ( n, F ; A )] = o ( n ) as n →∞ . Consequently, the AIPW estimator (8) may suffer from an efficiency loss which depends on how well the unknown treatment effect µ ∗ can be approximated by a member of the function class F ⊆ ( X × A → R ) under the ∥·∥ ( n ) -norm. Hence, any contribution to the MSE bound of the AIPW estimator (8) in addition to the efficient variance v 2 ∗ primarily relies on the approximation error associated with approximating the treatment effect µ ∗ utilizing a provided function class F .

3.5 Consequences for particular outcome models

The main goal of this section is to illustrate the consequences of our general theory developed in Section 3 so far for several concrete classes of outcome models. Throughout this section, we consider the case for which Y = [ -L, L ] for some constant L ∈ (0 , + ∞ ) , and impose the following condition:

Assumption 1 (Strict overlap condition) . The likelihood ratios are uniformly bounded by a universal constant B ∈ (0 , + ∞ ) , i.e., for every i ∈ [ n ] ,

We note that Assumption 1 is often referred to as the strict overlap condition in the literature of causal inference [20, 32, 66, 36, 11]. At this point, we emphasize that Assumption 1 is necessary to produce main consequences of the oracle inequality for the class of AIPW estimators (Theorem 3.2) that we discuss in the ensuing subsections: Theorems 3.3, 3.4, and the arguments throughout Appendix B.6.

3.5.1 Tabular case of the outcome model

We embark on our discussion about the consequences of our theory established in Sections 3.3 and 3.4 for one of the simplest case of the outcome model satisfying the following assumption.

Assumption 2 (Tabular setting of the outcome model) . The state-action space X × A is a finite set.

If we compute the gradient of the loss function (14), we have

where δ ( X i ,A i ) ∈ R X × A is the point-mass vector at the i -th state-action pair in the sample trajectory, i.e., δ ( X i ,A i ) ( x, a ) := 1 if ( x, a ) = ( X i , A i ) ; δ ( X i ,A i ) ( x, a ) := 0 otherwise.

Algorithm 3 Online gradient descent (OGD) algorithm for the finite state-action space.

Require: the function class F ⊆ [ -L, L ] X × A , the total number of rounds n ∈ N , and a sequence of learning rates { η i ∈ (0 , + ∞ ) : i ∈ [ n -1] } .

1: We first choose an initial point ˆ µ 1 ( ∅ ) ∈ F arbitrarily;

2: for i = 1 , 2 , · · · , n -1 , do

3: Observe a triple ( X i , A i , Y i ) ∈ O ;

4: Update ˆ µ i +1 ( O i ) ∈ F according to the following OGD update rule:

where Π F [ · ] : R X × A →F denotes the projection map of R X × A onto the function space F .

5: end for

6: return the sequence of estimates { ˆ µ i ( O i -1 ) ∈ F : i ∈ [ n ] } of the treatment effect µ ∗ .

Now, it is time to put forward an online contextual learning algorithm aimed at producing a sequence of estimates of µ ∗ with a no-regret learning guarantee. For the tabular case, the online non-parametric

regression problem can be resolved through standard online convex optimization (OCO) algorithms. In particular, we employ the online gradient descent (OGD) algorithm (see Algorithm 3) as a subroutine of Algorithm 1. By leveraging standard results on regret analysis of OCO algorithms, one can obtain the following regret bound, which guarantees a no-regret learning dynamics of Algorithm 3.

Theorem 3.3 (Regret guarantee of Algorithm 3) . Under Assumptions 1 and 2, the OGD algorithm (Algorithm 3) with learning rates { η i := diam ( F ) 4 LB 2 √ i : i ∈ [ n ] } guarantees

where diam ( F ) := sup {∥ µ ∥ 2 : µ ∈ F} denotes the diameter of F ⊆ [ -L, L ] X × A .

See Appendix B.4 for the proof of Theorem 3.3. Combining the regret guarantee (19) of Algorithm 3 together with the MSE bound (15) in Theorem 3.2, one can establish a concrete upper bound on the MSE of the AIPW estimator (8) by utilizing Algorithm 3 to produce a sequence of estimates for the treatment effect µ ∗ .

3.5.2 Linear function approximation

We next move on to outcome models where the state-action space X × A can be infinite. We begin with the simplest case: the class of linear outcome functions. Let ϕ ( · , · ) : X × A → R d be a known feature map such that sup {∥ ϕ ( x, a ) ∥ 2 : ( x, a ) ∈ X × A } ≤ 1 , and we consider the functions that are linear in this representation: f θ ( · , · ) : X × A → R , where f θ ( x, a ) := θ ⊤ ϕ ( x, a ) for some parameter vector θ ∈ R d . Given a radius R > 0 , we define the function class

where B ( 0 d ; R ) := { u ∈ R d : ∥ u ∥ 2 ≤ R } . With this linear function approximation framework, let us consider the following OCO model: at the i -th stage,

(i) the learner first chooses a point ˆ θ i ( O i -1 ) ∈ Θ ;

(ii) the environment then picks a loss function L i ( · ) : Θ → R defined as

and our goal is to produce a sequence of estimates { ˆ µ i ( O i -1 ) := { ˆ θ i ( O i -1 ) } ⊤ ϕ ∈ F lin : i ∈ [ n ] }

for the treatment effect µ ∗ after n rounds of the above-mentioned OCO model which minimizes the learner's regret against the best fixed action in hindsight :

where A is the learner's OCO algorithm whose output is a sequence { ˆ θ i ( O i -1 ) ∈ Θ : i ∈ [ n ] } of parameters. If we compute the gradient of the loss function (21), one has

For the current linear function approximation setting, we implement the OGD algorithm (Algorithm 4) as a sub-routine of Algorithm 1. By using the same arguments as in Section 3.5.1, one can reproduce the following regret guarantee of Algorithm 4 whose proof is available at Appendix B.5.

Theorem 3.4 (Regret guarantee of Algorithm 4) . With Assumption 1, the OGD algorithm (Algorithm 4) with learning rates { η i := R B 2 ( L + R ) √ i : i ∈ [ n ] } guarantees

Algorithm 4 Online gradient descent (OGD) algorithm for linear function approximation.

Require: the radius R > 0 of the parameter space, the number of rounds n ∈ N , and a sequence of learning rates { η i ∈ (0 , + ∞ ) : i ∈ [ n -1] } .

1: We first choose an arbitrary initial point ˆ θ 1 ( ∅ ) ∈ Θ , where Θ := B ( 0 d ; R ) ;

2: for i = 1 , 2 , · · · , n -1 , do

3: Observe a triple ( X i , A i , Y i ) ∈ O ;

4: Update ˆ θ i +1 ( O i ) ∈ Θ according to the following OGD update rule:

where Π Θ [ · ] : R d → Θ denotes the projection map of R d onto the parameter space Θ .

5: end for

6: return the estimates { ˆ µ i ( O i -1 ) := { ˆ θ i ( O i -1 ) } ⊤ ϕ ∈ F lin : i ∈ [ n ] } of the treatment effect.

General function approximation Lastly, we demonstrate the consequences of our general theory established in Sections 3.3 and 3.4 for the case of general function approximation: the function class F ⊆ ( X × A → [ -L, L ]) can be arbitrarily chosen. Our further discussion this case heavily relies on the basic theory of online non-parametric regression from [45] whose technical details are rather long and complicated. So, we defer our detailed inspection on the case of general function approximation to Appendix B.6.

4 Lower bounds: local minimax risk

We turn our attention to a local minimax lower bound for estimating the off-policy value τ ∗ = τ ( I ∗ ) . Here, we aim at establishing lower bounds that hold uniformly over all estimators that are permitted to know both the propensity scores { π ∗ i ( X i , O i -1 ; A i ) : i ∈ [ n ] } and the evaluation function g . We assume the existence of a constant K ≥ 1 and reference Markov policies { Π i : X → ∆( A ) : i ∈ [ n ] } such that Π i ( ·| x ) ≪ λ A ( · ) for ( x, i ) ∈ X × [ n ] , and

for all ( x, o i -1 , a ) ∈ X × O i -1 × A , where π i ( x, · ) := dΠ i ( ·| x ) d λ A : A → R + for each context x ∈ X . Proximity of behavioral policies to certain Markov policies is often assumed under adaptive data collection procedures. For instance, in Theorem 1 of [67], the authors assumed that the sequence of behavior policies is eventually Markov ; see the equation (8) therein.

4.1 Instance-dependent local minimax lower bounds

Given any problem instance I ∗ = (Ξ ∗ , Γ ∗ ) ∈ I and an error function δ : X × A → R + , we consider the following local neighborhoods:

where for any given Γ : X × A → ∆( Y ) , let µ (Γ)( x, a ) := ∫ Y y Γ(d y | x, a ) for each ( x, a ) ∈ X × A . Our goal is to lower bound the following local minimax risk :

where C δ ( I ∗ ) := N (Ξ ∗ ) ×N δ (Γ ∗ ) ⊆ I . We now specify some assumptions necessary for lower bounding the local minimax risk (26). Prior to this, we introduce a new important notation: given any random variable Y ∈ L 4 (Ω , F , P ) defined on a probability space (Ω , F , P ) , its (2 , 4) -moment ratio is defined as ∥ Y ∥ 2 → 4 := √ E [ Y 4 ] E [ Y 2 ] .

Assumption 3. Let h ( x ) := ⟨ g ( x, · ) , µ ∗ ( x, · ) ⟩ λ A -E X ∼ Ξ ∗ [ ⟨ g ( X, · ) , µ ∗ ( X, · ) ⟩ λ A ] . We assume that H 2 → 4 := ∥ h ∥ 2 → 4 = √ E X ∼ Ξ ∗ [ h 4 ( X )] E X ∼ Ξ ∗ [ h 2 ( X )] < + ∞ .

We next make an assumption on a lower bound on the local neighborhood size :

Assumption 4. The neighborhood function δ ( · , · ) : X × A → R + satisfies the lower bound

for all ( x, a, i ) ∈ X × A × [ n ] .

We note that Assumptions 3 and 4 are analogues of Assumptions (MR) and (LN) considered in [42], respectively, for the case of adaptively collected data. Under these assumptions, one can prove the following lower bound on the local minimax risk over C δ ( I ∗ ) :

Theorem 4.1. Under Assumptions 3 and 4, the local minimax risk over C δ ( I ∗ ) is lower bounded by

where C ( K ) > 0 is a universal constant that only depends on the data coverage constant K ≥ 1 of the reference Markov policies { Π i ( · ) : X → ∆( A ) : i ∈ [ n ] } defined in (25) .

The proof of Theorem 4.1 can be found in Appendix C.1. This result delivers a key message: the term v 2 ∗ n including the sequentially weighted ℓ 2 -norm is indeed the fundamental limit for estimating the linear functional based on adaptively collected data. Our results can be viewed as a generalization of those developed in [42] for the case of i.i.d. data.

Acknowledgments and Disclosure of Funding

Jeonghwan Lee was partially supported by the Kwanjeong Educational Foundation. Cong Ma was partially supported by the National Science Foundation via grant DMS-2311127.

References

Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 687-696, 2017.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Timothy B Armstrong and Michal Kolesár. Finite-sample optimal estimation and inference on average treatment effects under unconfoundedness. Econometrica , 89(3):1141-1177, 2021.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whitney Newey. Double/debiased/neyman machine learning of treatment effects. American Economic Review , 107(5):261-265, 2017.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.

Thomas Cook, Alan Mishler, and Aaditya Ramdas. Semiparametric efficient inference in adaptive experiments. In Causal Learning and Reasoning , pages 1033-1064. PMLR, 2024.

Jessica Dai, Paula Gradu, and Christopher Harshaw. Clip-ogd: An experimental design for adaptive neyman allocation in sequential experiments. Advances in Neural Information Processing Systems , 36, 2024.

Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.

Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601 , 2011.

Alexander D'Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. Overlap in observational studies with high-dimensional covariates. Journal of Econometrics , 221(2):644654, 2021.

Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning , pages 1447-1456. PMLR, 2018.

Nicolò Felicioni, Maurizio Ferrari Dacrema, Marcello Restelli, and Paolo Cremonesi. Offpolicy evaluation with deficient support using side information. Advances in Neural Information Processing Systems , 35:30250-30264, 2022.

Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review , 70(3):419-435, 2002.

Evarist Giné and Joel Zinn. Some limit theorems for empirical processes. The Annals of Probability , pages 929-989, 1984.

Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments.' arxiv e-prints. arXiv preprint arXiv:1911.02768 , 2019.

Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the national academy of sciences , 118(15):e2014602118, 2021.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Tim Hesterberg. Weighted average importance sampling and defensive mixture distributions. Technometrics , 37(2):185-194, 1995.

Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica , 71(4):1161-1189, 2003.

Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663-685, 1952.

Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics , 49(2):1055 1080, 2021.

Guido W Imbens. Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and statistics , 86(1):4-29, 2004.

Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statistics , 17(2):295-311, 2008.

Ying Jin, Zhimei Ren, Zhuoran Yang, and Zhaoran Wang. Policy learning" without'overlap: Pessimism and generalized empirical bernstein's inequality. arXiv preprint arXiv:2212.09900 , 2022.

Nathan Kallus, Yuta Saito, and Masatoshi Uehara. Optimal off-policy evaluation from multiple logging policies. In International Conference on Machine Learning , pages 5247-5256. PMLR, 2021.

Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In International conference on artificial intelligence and statistics , pages 1243-1251. PMLR, 2018.

Joseph DY Kang and Joseph L Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. 2007.

Nikos Karampatziakis, Paul Mineiro, and Aaditya Ramdas. Off-policy confidence sequences. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2021.

Masahiro Kato, Takuya Ishihara, Junya Honda, and Yusuke Narita. Efficient adaptive experimental design for average treatment effect estimation. arXiv preprint arXiv:2002.05308 , 2020.

Shakeeb Khan and Elie Tamer. Irregular identification, support conditions, and inverse weight estimation. Econometrica , 78(6):2021-2042, 2010.

Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Haanvid Lee, Jongmin Lee, Yunseon Choi, Wonseok Jeon, Byung-Jun Lee, Yung-Kyun Noh, and Kee-Eung Kim. Local metric learning for off-policy evaluation in contextual bandits with continuous actions. Advances in Neural Information Processing Systems , 35:3913-3925, 2022.

Lihua Lei, Alexander D'Amour, Peng Ding, Avi Feller, and Jasjeet Sekhon. Distribution-free assessment of population overlap in observational studies. Technical report, Working paper, Stanford University, 2021.

Gen Li and Weichen Wu. Sharp high-probability sample complexities for policy evaluation with linear function approximation. arXivorg , 2023.

Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estimation. In Artificial Intelligence and Statistics , pages 608-616. PMLR, 2015.

Roderick JA Little and Donald B Rubin. Statistical analysis with missing data , volume 793. John Wiley & Sons, 2019.

Alexander R Luedtke and Mark J Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics , 44(2):713, 2016.

Cong Ma, Banghua Zhu, Jiantao Jiao, and Martin J Wainwright. Minimax off-policy evaluation for multi-armed bandits. IEEE Transactions on Information Theory , 68(8):5314-5339, 2022.

Wenlong Mou, Martin J Wainwright, and Peter L Bartlett. Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency. arXiv preprint arXiv:2209.13075 , 2022.

Yusuke Narita, Shota Yasui, and Kohei Yata. Efficient counterfactual learning from bandit feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4634-4641, 2019.

Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang, Jian Pei, and Peng Cui. Offline policy evaluation in large action spaces via outcome-oriented action grouping. In Proceedings of the ACM Web Conference 2023 , pages 1220-1230, 2023.

Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264. PMLR, 2014.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability theory and related fields , 161:111-153, 2015.

Sasha Rakhlin, Ohad Shamir, and Karthik Sridharan. Relax and randomize: From value to algorithms. Advances in Neural Information Processing Systems , 25, 2012.

James Robins, Mariela Sued, Quanhong Lei-Gomez, and Andrea Rotnitzky. Comment: Performance of double-robust estimators when" inverse probability" weights are highly variable. Statistical Science , 22(4):544-559, 2007.

James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122-129, 1995.

James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846-866, 1994.

James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the american statistical association , 90(429):106-121, 1995.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. arXiv preprint arXiv:2202.06317 , 2022.

Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. In international conference on Machine learning , pages 29734-29759. PMLR, 2023.

Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit exploration data. Advances in neural information processing systems , 23, 2010.

Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning , pages 9167-9176. PMLR, 2020.

Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning , pages 6005-6014. PMLR, 2019.

Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems , 28, 2015.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Mark J van der Laan. The construction and analysis of adaptive group sequential designs. 2008.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Lequn Wang, Akshay Krishnamurthy, and Alex Slivkins. Oracle-efficient pessimism: Offline policy optimization in contextual bandits. In International Conference on Artificial Intelligence and Statistics , pages 766-774. PMLR, 2024.

Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudık. Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning , pages 3589-3597. PMLR, 2017.

Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytime-valid off-policy inference for contextual bandits. ACM/JMS Journal of Data Science , 1(3):1-42, 2024.

S Yang and P Ding. Asymptotic inference of causal effects with observational studies trimmed by the estimated propensity scores. Biometrika , 105(2):487-493, 03 2018.

Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125-2135, 2021.

Ruohan Zhan, Zhimei Ren, Susan Athey, and Zhengyuan Zhou. Policy learning with adaptively collected data. Management Science , 2023.

Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. Advances in neural information processing systems , 33:9818-9829, 2020.

A Some elementary inequalities and their proofs

The following lemma is useful for the truncation arguments used in the proofs of our local minimax lower bounds. In particular, it enables to make small modifications on a pair of probability measures by conditioning on good events of each probability measure, without inducing an overly large change in the total variation distance.

Lemma A.1. Let ( µ, ν ) be a pair of probability measures defined on a common sample space (Ω , F ) , and consider any two events A,B ∈ F satisfying min { µ ( A ) , ν ( B ) } ≥ 1 -ϵ for some ϵ ∈ [ 0 , 1 4 ] . Then, the conditional distributions ( µ | A ) ( · ) ∈ ∆(Ω , F ) and ( ν | B ) ( · ) ∈ ∆(Ω , F ) defined by

for any event E ∈ F , satisfy the bound

Proof of Lemma A.1. Due to the triangle inequality for the total variation (TV) distance, it follows that

and

At this point, one can easily observe that

Putting the observation (32) into the inequalities (30) and (31), the assumptions 1 -µ ( A ) ≤ ϵ and 1 -ν ( B ) ≤ ϵ establish the desired result.

B Proofs and omitted details for Section 3

B.1 Proof of Proposition 3.1

First, one can observe that

We now assume (3) and note that

One can reveal that

where the step (a) can be verified as follows:

Next, we compute Cov I ∗ [ g ( X i ,A i ) Y i π ∗ i ( X i , O i -1 ; A i ) -f i ( X i , O i -1 , A i ) , g ( X j ,A j ) Y j π ∗ j ( X j , O j -1 ; A j ) -f j ( X j , O j -1 , A j ) ] :

where the step (b) holds due to the fact that X j is independent of the historical data H j -1 , which immediately yields X j | H j -1 d = X j ∼ Ξ ∗ ( · ) . Taking two pieces (35) and (36) collectively into the equation (34), one has

as desired.

B.2 Proof of Theorem 3.1

We first single out a key technical lemma throughout this section that plays a crucial role in the proof of Theorem 3.1.

Lemma B.1. The following results hold:

Proof of Lemma B.1.

(i) From the definition of ˆ Γ i ( · ) : O i → R in (9), we have

Thus, we obtain

as desired.

(ii) One can reveal that

where the step (a) holds due to the facts that ˆ Γ i ( O i ) is H j -1 -measurable and X j ⊥ ⊥ H j -1 , together with the equation (37).

(iii) It follows that

as desired, where the step (b) follows from the fact (40).

Now, it's time to finish the proof of Theorem 3.1. One can reveal that

where the step (a) holds from the part (ii) of Lemma B.1, the step (b) makes use of the inequality (38), and the step (c) follows from the definition of v 2 ∗ in (7).

B.3 Proof of Theorem 3.2

It holds due to the observation (12) that

which establishes the following expression of the estimation error term (11):

At this point, one can realize that

where the step (a) holds by the fact (12). Taking two pieces (43) and (44) collectively, it follows that

Hence, the upper bound (15) on the MSE of the AIPW estimator (8) is an immediate consequence of the inequality (45) by putting it into the bound (10) in Theorem 3.1.

B.4 Proof of Theorem 3.3

One can easily observe from the equation (17) for every µ ∈ F that

which holds due to Assumption 1 together with the fact Y = [ -L, L ] . So it turns out that the loss function (14) is Lipschitz continuous with parameter G := 4 LB 2 P n I ∗ -almost surely. Hence, the desired conclusion immediately follows by Theorem 3.1 in [18] with parameter G = 4 LB 2 .

B.5 Proof of Theorem 3.4

One can realize from the equation (22) that P n I ∗ -almost surely,

which holds by Assumption 1 together with the facts Y = [ -L, L ] and sup ( x,a ) ∈ X × A ∥ ϕ ( x, a ) ∥ 2 ≤ 1 . So, the loss function (21) is Lipschitz continuous with parameter G := 2 B 2 ( L + R ) P n I ∗ -a.s. Hence, the desired result follows by Theorem 3.1 in [18] with parameter G = 2 B 2 ( L + R ) and D = 2 R .

B.6 Consequences for particular outcome models: general function approximation

Lastly, we consider the most challenging setting where the estimation of the treatment effect µ ∗ ( · , · ) : X × A → R is parameterized by general function classes. Under Assumption 1, one first observes from the MSE bound (10) of the AIPW estimator (8) in Theorem 3.1 that

From the last term in the MSE bound (48), our aim becomes to control an upper bound of the term

in the finite sample regime. Towards achieving this goal, we consider the online non-parametric regression problem described in Algorithm 2 whose sequence { l i ( · ) : ( X × A → R ) → R : i ∈ [ n ] } of loss functions defined as (14) is superseded by { l i ( · ) : ( X × A → R ) → R : i ∈ [ n ] } , where

It is straightforward to see for every i ∈ [ n ] that

With this modified online non-parametric regression problem, we now aim to minimize the learner's modified regret defined as follows:

where A denotes the learner's online non-parametric regression algorithm that returns a sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect based on interactions with the environment which selects modified loss functions { l i ( · ) : ( X × A → R ) → R : i ∈ [ n ] } .

Theorem B.1. The AIPW estimator (8) based on a sequence { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of estimates for the treatment effect µ ∗ produced by making use of an online non-parametric regression algorithm A against the environment which chooses the sequence of modified loss functions

Proof of Theorem B.1. It follows from the property (51) that

which leads to the following expression of the estimation error term (49):

Here, one may observe that

where the step (a) holds by the fact (51). Putting two pieces (54) and (56) together yields

Hence, the desired result (53) on the MSE of the AIPW estimator (8) is a straightforward consequence of the inequality (57) by plugging it into the bound (48).

Here, we remark that aside from the optimal variance v 2 ∗ , the MSE bound (53) shows two additional terms: (i) the expected regret relative to the number of rounds n , where the expectation is taken over O n ∼ P n I ∗ ( · ) ; and (ii) the approximation error term whose form is slightly different from the one inf { ∥ µ -µ ∗ ∥ 2 ( n ) : µ ∈ F } appeared in the MSE bound (15) of Theorem 3.2.

Non-asymptotic theory of online non-parametric regression Before delving into the investigation of the modified regret (52), we briefly recap the main results in [45] that establishes a theoretical framework of online non-parametric regression. In contrast to most existing works of online regression, the authors do NOT start from an algorithm, but instead directly work with the minimax regret in [45]. We will be able to extract a (not necessarily efficient) algorithm after taking a closer look at the minimax regret. Let us use ⟪ · · · ⟫ n i =1 to denote an interleaved application of the operators inside repeated over n rounds. With this notation in hand, the minimax regret of the online non-parametric regression problem for estimation of the treatment effect can be written as

where F ⊆ ( X × A → [ -L, L ]) is a pre-specified function class. One of the key tools in the study of estimators based on i.i.d. data is the symmetrization technique [15, 62]. Under the i.i.d. scenario, one can investigate the supremum of an empirical process conditionally on the data by introducing Rademacher random variables, which is NOT directly applicable given the adaptive nature of our main problem. In the online prediction scenario, such a symmetrization technique becomes more subtle and it requires the notion of a binary tree, the smallest entity which captures the sequential nature of the problem in some sense. Towards achieving our goal in our problem, let us state some definitions.

Definition B.1. Let S be a measurable state space. An S -valued tree of depth n is a rooted complete binary tree with nodes labeled by elements of the state space S : the sequence s = ( s 1 , s 2 , · · · , s n ) of labeling functions s i ( · ) : {± 1 } i -1 → S which provides the labels of each node. Here, s 1 ∈ S is the label for the root of the tree , while s i for 2 ≤ i ≤ n is the label of the node obtained by following the path of length i -1 from the root, with +1 indicating right and -1 indicating left . A path of length n is given by the sequence ϵ 1: n = ( ϵ 1 , · · · , ϵ n ) ∈ {± 1 } n . Given any measurable function ϕ ( · ) : S → R , ϕ ( s ) is an R -valued tree of depth n with labeling functions ( ϕ ◦ s i ) ( · ) : {± 1 } i -1 → R for level i ∈ [ n ] (or, in words, the evaluation of ϕ ( · ) : S → R , ϕ ( s ) on s ). Lastly, we let Tree ( S , n ) denote the set of all S -valued trees of depth n .

Here, one may think of the sequence of functions { s i ( · ) : i ∈ [ n ] } defined on the underlying sample space as a predictable stochastic process with respect to the dyadic filtration { σ ( ϵ 1: i ) : i ∈ [ n ] } . Next, let us define the notion of a sequential β -cover quantifies one of the key complexity measures of a function class G ⊆ ( S → R ) evaluated on the predictable process: the sequential covering number .

Definition B.2 (Sequential covering numbers [46]) .

(i) Define the following random pseudo-metric between two R -valued trees u = ( u i : i ∈ [ n ]) and v = ( v i : i ∈ [ n ]) of depth n : for any ( p, ϵ 1: n ) ∈ [1 , + ∞ ] ×{± 1 } n ,

(ii) Aset V ⊆ Tree ( R , n ) is called a sequential β -cover with respect to l p -norm of G ⊆ ( S → R ) on a given S -valued tree s of depth n , where p ∈ [1 , + ∞ ] , if

where G ( s ) := { g ( s ) : g ∈ G} ⊆ Tree ( R , n ) ;

(iii) The sequential β -covering number with respect to l p -norm of a function class G ⊆ ( S → R ) on an S -valued tree s of depth n , where p ∈ [1 , + ∞ ] , is defined by

:= min {| V | : V ⊆ Tree ( R , n ) is a sequential β -cover w.r.t. l p -norm of G on s } .

Let us further define N p ( β, G , n ) := sup {N p ( β, G , s ) : s ∈ Tree ( S , n ) } to be the maximal sequential β -covering number with respect to l p -norm of G over S -valued trees of depth n . Now, we will refer to log N p ( β, G , n ) as the sequential β -metric entropy of G with respect to l p -norm .

In particular, we are going to study the behavior of the minimax regret V n ( F ) for the case where the sequential metric entropy of F ⊆ ( X × A → [ -L, L ]) w.r.t. l 2 -norm grows polynomially as the scale β decreases:

Let us also consider the parametric ' p = 0 ' case when the sequential covering number of F with respect to l 2 -norm itself behaves as:

For instance, the function class F := { f θ ( · ) : R d → R : θ ∈ Θ } for the linear regression problem in a bounded measurable subset Θ ⊆ R d , where the function f θ ( · ) : R d → R is given by f θ ( x ) := θ ⊤ x for θ ∈ R d , satisfies the condition (62). By employing the main results (in particular, Theorem 2 ) in [45], one can establish the following conclusion:

Theorem B.2 (The rates of convergence of the minimax regret) . Given any function class F ⊆ ( X × A → [ -L, L ]) with sequential metric entropy growth log N 2 ( β, F , n ) ≤ β -p for p ∈ (0 , + ∞ ) , it holds that

(i) for p ∈ (2 , + ∞ ) , the minimax regret (58) is bounded as

(ii) for p ∈ (0 , 2) , the minimax regret (58) is bounded as

(iii) for p = 2 , the minimax regret (58) is bounded as

(iv) for the parametric case (62) , the minimax regret (58) is bounded as

(v) if the function class F ⊆ ( X × A → [ -L, L ]) is a finite set, the minimax regret (58) is bounded as

It is shown in [45] that the upper bounds (i)-(iv) on the minimax regret (58) in Theorem B.2 are tight up to logarithmic factors . See Theorem 3 therein for further details.

Although Theorem B.2 characterizes the rates of convergence of the minimax regret (58) in various scenarios statistically , its proof is non-constructive in the sense that the regret bounds therein are established without explicitly constructing an algorithm. In order to provide a general algorithmic framework for the problem of online non-parametric regression, we follow the abstract relaxation recipe proposed in [47]. It was shown in [47] that if one can find a sequence of mappings from the observed data to real numbers Rel n , often called a relaxation , satisfying some desirable conditions, then one can construct estimators based on such relaxations. To be specific, we search for a relaxation Rel n ( · , · ) : ⊎ n k =0 { ( X × A ) k × [ -L, L ] k } → R that satisfies the following two conditions:

Assumption 5 (Initial condition) . The relaxation Rel n ( · , · ) : ⊎ n k =0 { ( X × A ) k × [ -L, L ] k } → R satisfies

where ( x , a ) 1: k := (( x i , a i ) : i ∈ [ k ]) ∈ ( X × A ) k and y 1: k := ( y i : i ∈ [ k ]) ∈ [ -L, L ] k for every k ∈ [ n ] .

Assumption 6 (Recursive admissibility condition) . The relaxation Rel n ( · , · ) satisfies

for any k ∈ [ n ] and any x k ∈ X .

A relaxation Rel n ( · , · ) : ⊎ n k =0 { ( X × A ) k × [ -L, L ] k } → R satisfying Assumptions 5 and 6 is said to be admissible . With an admissible relaxation Rel n ( · , · ) in hand, one can design an algorithm for the online non-parametric regression problem with the following associated regret bound (see Algorithm 5 for a detailed description):

We further notice that if the function y i ∈ [ -L, L ] ↦→ (ˆ y -y i ) 2 + Rel n ((( x , a ) 1: i ) , ( y 1: i -1 , y i )) is convex for every (ˆ y, x 1: n , a 1: n , y 1: i -1 ) ∈ [ -L, L ] × X n × A n × [ -L, L ] i -1 and i ∈ [ n ] , then the prediction rules (71) and (72) becomes much simpler, since the supremum over y i ∈ [ -L, L ] is attained either L or -L . The prediction rules then can be written as

and for i ∈ { 2 , 3 , · · · , n } ,

One can easily observe that the prediction rules (73) and (74) can be further simplified as

Algorithm 5 A generic forecaster based on the relaxation recipe proposed in [47]

1: We first choose ˆ µ 1 ( ∅ )( · , · ) ∈ ( X × A → R ) as

2: for i = 2 , 3 , · · · , n , do

3: Observe a triple ( X i , A i , Y i ) ∈ O ;

4: We compute ˆ µ i ( O i -1 ) ∈ ( X × A → R ) according to the following rule:

5: end for

6: return the sequence of estimates { ˆ µ i ( O i -1 ) ∈ ( X × A → R ) : i ∈ [ n ] } of the treatment effect.

where χ [ -L,L ] ( · ) : R → [ -L, L ] defines a clip function onto the interval [ -L, L ] , i.e.,

By directly using Lemma 16 in [45], one can obtain the following significant result:

Theorem B.3. The relaxation R n ( · , · ) : ⊎ n k =0 { ( X × A ) k × [ -L, L ] k } → R defined as

where the pair ( z , m ) ranges over the set Tree ( X × A , n ) × Tree ( R , n ) , is an admissible relaxation. As a direct consequence of the regret bound (70) , Algorithm 5 using the admissible relaxation R n ( · , · ) as an input enjoys the regret bound of an offset Rademacher complexity:

Since the upper bounds on the minimax regret (58) provided in Theorem B.2 are established by further upper bounding the offset Rademacher complexity R n ( ∅ , ∅ ) , one can end up with the following corollary:

Corollary B.1. Consider any function class F ⊆ ( X × A → [ -L, L ]) with sequential metric entropy growth log N 2 ( β, F , n ) ≤ β -p for p ∈ (0 , + ∞ ) . Then, Algorithm 5 using the admissible relaxation R n ( · , · ) defined by (77) as an input enjoys the following regret bounds:

(i) for p ∈ (2 , + ∞ ) , it holds that

(ii) for p ∈ (0 , 2) , it holds that

(iii) for p = 2 , it holds that

(iv) for the parametric case (62) , it holds that

(v) if the function class F ⊆ ( X × A → [ -L, L ]) is a finite set, it holds that

Even though Corollary B.1 gives no-regret learning guarantees of Algorithm 5 with the admissible relaxation R n ( · , · ) defined by (77) for various function classes F ⊆ ( X × A → [ -L, L ]) , it is still NOT a practical algorithm since the relaxation R n ( · , · ) defined as (77) is not directly computable in general. To address this problem, [45] provided a generic schema for deriving implementable online non-parametric regression algorithms. The schema can be described as follows:

(a) Find a computable relaxation Rel n ( · , · ) : ⊎ n k =0 { ( X × A ) k × [ -L, L ] k } → R such that

for every ( k, x 1: n , a 1: n , y 1: n ) ∈ { 0 , 1 , · · · , n } × X n × A n × [ -L, L ] n , and the function y k ∈ [ -L, L ] ↦→ (ˆ y -y k ) 2 + Rel n ((( x , a ) 1: k ) , ( y 1: k -1 , y k )) ∈ R is convex for every (ˆ y, x 1: n , a 1: n , y 1: k -1 ) ∈ [ -L, L ] × X n × A n × [ -L, L ] k -1 and k ∈ [ n ] ;

(b) Next, we check the following condition:

(c) Implement Algorithm 5 using the relaxation Rel n ( · , · ) as an input.

The authors proved that any computable relaxation Rel n ( · , · ) satisfying conditions stated in (a) and (b) are admissible; see Proposition 17 therein. Consequently, any online non-parametric regression algorithm produced by the above generic schema always satisfies the regret bound (70). Moreover, the authors established a practical online non-parametric regression algorithm with no-regret learning guarantees based on the above schema for the finite function class F ⊆ ( X × A → [ -L, L ]) and the online linear regression problem.

C Proofs for Section 4

C.1 Proof of Theorem 4.1

Theorem 4.1 can be established by taking the following two lemmas collectively:

Lemma C.1. Under Assumption 3, the local minimax risk over the class C δ ( I ∗ ) is lower bounded by

provided that n ≥ 16 H 2 2 → 4 .

Lemma C.2. Under Assumption 4, the local minimax risk over the class C δ ( I ∗ ) is lower bounded by

C.2 Proof of Lemma C.1

The proof relies on Le Cam's two-point method by taking the outcome kernel Γ ∗ : X × A → ∆( Y ) to be fixed, and perturbing the context distribution Ξ ∗ ( · ) ∈ ∆( X ) : we first construct a collection of context distributions { Ξ s ( · ) ∈ ∆( X ) : s ∈ (0 , + ∞ ) } . Later, we will choose the parameter s > 0 small enough so that Ξ s ∈ N (Ξ ∗ ) and two distributions P n (Ξ s , Γ ∗ ) ∈ ∆( O n ) and P n (Ξ ∗ , Γ ∗ ) ∈ ∆( O n ) are indistinguishable , but large enough such that the functional values τ (Ξ s , Γ ∗ ) and τ (Ξ ∗ , Γ ∗ ) are well-separated . Le Cam's two-point lemma (the equation (15.14) in [62]) guarantees that the local minimax risk M n ( C δ ( I ∗ )) is lower bounded as

provided that Ξ s ∈ N (Ξ ∗ ) .

As the first step, we upper bound the total variation distance TV ( P n (Ξ s , Γ ∗ ) , P n (Ξ ∗ , Γ ∗ ) ) . Thanks to the Pinsker-Csiszár-Kullback inequality, one has

We can find that the density function of the law P n I = P n (Ξ , Γ) ∈ ∆( O n ) of the sample trajectory O n under the problem instance I = (Ξ , Γ) ∈ I with respect to the base measure ( λ X ⊗ λ A ⊗ λ A ) ⊗ n is given by

Using this fact, the KL-divergence KL ( P n (Ξ s , Γ ∗ ) ∥ ∥ ∥ P n (Ξ ∗ , Γ ∗ ) ) can be computed as

So if one can show that Ξ s ∈ N (Ξ ∗ ) , then the equation (89) guarantees that

which can be taken collectively with the bound (87) to produce the following conclusion:

With the arguments thus far in place, it remains to construct a family { Ξ s ∈ ∆( X ) : s ∈ (0 , + ∞ ) } and then choose a parameter s > 0 such that Ξ s ∈ N (Ξ ∗ ) and the functional values τ (Ξ s , Γ ∗ ) and τ (Ξ ∗ , Γ ∗ ) are well-separated. To this end, we consider the function ˜ h ( · ) : X → R defined by

Since H 2 → 4 ≥ 1 , one can easily find that ∣ ∣ ∣ ˜ h ( x ) ∣ ∣ ∣ ≤ | h ( x ) | for all x ∈ X . Now for each s ∈ (0 , + ∞ ) , we define the tilted probability measure Ξ s ( · ) ∈ ∆( X ) by

where Z ( s ) := ∫ X ξ ∗ ( x ) exp ( s ˜ h ( x ) ) d λ X ( x ) = E X ∼ Ξ ∗ [ exp ( s ˜ h ( X ) )] . At this point, we note for every x ∈ X that

which also immediately yields

Here, we choose s = 1 4 ∥ h ∥ L 2 (Ξ ∗ ) √ n > 0 . Then, it holds due to the fact ∣ ∣ ∣ ˜ h ( x ) ∣ ∣ ∣ ≤ 2 H 2 → 4 ∥ h ∥ L 2 (Ξ ∗ ) for all x ∈ X that

where the step (a) follows due to the assumption that n ≥ 16 H 2 2 → 4 . Now, it's time to prove that Ξ s ∈ N (Ξ ∗ ) for the current choice of the parameter s > 0 . Due to Theorem 5 in [14], it follows that

So it suffices to upper bound the χ 2 -divergence χ 2 (Ξ s ∥ Ξ ∗ ) . One can reveal that

where the step (b) makes use of the fact (93) together with the elementary bound | exp( u ) -1 | ≤ | u | exp( | u | ) , ∀ u ∈ R , and the step (c) follows from the fact ∣ ∣ ∣ ˜ h ( x ) ∣ ∣ ∣ ≤ | h ( x ) | , ∀ x ∈ X . If we put

s = 1 4 ∥ h ∥ L 2 (Ξ ∗ ) √ n into the bound (96), then we obtain from the fact s ∥ ∥ ∥ ˜ h ∥ ∥ ∥ ∞ ≤ 1 8 together with the basic inequality (95) that

which implies Ξ s ∈ N (Ξ ∗ ) for the choice of the parameter s = 1 4 ∥ h ∥ L 2 (Ξ ∗ ) √ n . Hence, the upper bound on the total variation distance (90) turns out to be valid.

Next, we lower bound the gap between the functional values τ (Ξ s , Γ ∗ ) and τ (Ξ ∗ , Γ ∗ ) . It holds that

Since s ∥ ∥ ∥ ˜ h ∥ ∥ ∥ ∞ ≤ 1 8 , we have s ˜ h ( X ) ∈ [ -1 4 , 1 4 ] and therefore the simple inequality

implies

where the step (d) holds due to the fact that sign ( h ( x )) = sign ( ˜ h ( x ) ) , ∀ x ∈ X , the step (e) makes use of the property that ∣ ∣ ∣ ˜ h ( x ) ∣ ∣ ∣ ≤ | h ( x ) | , ∀ x ∈ X , together with the Cauchy-Schwarz inequality, the step (f) follows due to Lemma 7 in [42], and the step (g) utilizes the assumption that n ≥ 16 H 2 2 → 4 . Putting the lower bound (99) into the equation (98) yields

where the step (h) holds since E X ∼ Ξ ∗ [ exp ( s ˜ h ( X ) )] ≤ 3 2 , which follows by the fact ∣ ∣ ∣ s ˜ h ( X ) ∣ ∣ ∣ ≤ 1 8 . Finally, by taking three pieces (86), (90), and (100) collectively, one completes the proof of Lemma C.1.

C.3 Proof of Lemma C.2

The proof of Lemma C.2 is also heavily relies on Le Cam's two-point method. For each ( i, s, z ) ∈ [ n ] × (0 , + ∞ ) ×{± 1 } , we consider the function µ i ( zs )( · , · ) : X × A → R defined by

Also, we define the perturbed outcome kernel Γ i ( zs )( · , · ) : X × A → Y as

Then, due to Le Cam's two-point lemma, the local minimax risk over the class C δ ( I ∗ ) can be lower bounded by

M n ( C δ ( I ∗ )) ≥ 1 4 { 1 -TV ( P n (Ξ ∗ , Γ i ( s )) , P n (Ξ ∗ , Γ i ( -s )) )} { τ (Ξ ∗ , Γ i ( s )) -τ (Ξ ∗ , Γ i ( -s )) } 2 , (102) provided that Γ i ( zs ) ∈ N δ (Γ ∗ ) for z ∈ {± 1 } .

We first upper bound the total variation distance TV ( P n (Ξ ∗ , Γ i ( s )) , P n (Ξ ∗ , Γ i ( -s )) ) . By employing the Pinsker-Csiszár-Kullback inequality, one has

Note that

By utilizing the fact (105), one can obtain from the equation (104) that

where the step (a) follows by the assumption (25). If we put s = 1 2 K √ n ∥ σ ∥ ( n ) into the bound (106), it follows that KL ( P n (Ξ ∗ , Γ i ( s )) ∥ ∥ ∥ P n (Ξ ∗ , Γ i ( -s )) ) ≤ 1 2 . So, by combining this conclusion together with the basic inequality (103), we arrive at

At this point, we should note for every ( i, z, x, a ) ∈ [ n ] ×{± 1 } × X × A that

where the step (b) holds due to Assumption 4, and the step (c) utilizes the fact that K ≥ 1 , which establishes that Γ i ( zs ) ∈ N δ (Γ ∗ ) for z ∈ {± 1 } and thus the local minimax lower bound (102) turns out to be valid.

Next, we aim at establishing a lower bound on the gap between the functional values τ (Ξ ∗ , Γ i ( s )) and τ (Ξ ∗ , Γ i ( -s )) . One can observe that

where the step (d) holds due to the assumption (25). By taking three pieces (102), (107), and (109) collectively, we have

for every i ∈ [ n ] . Hence, one can conclude by taking an average of the local minimax lower bound (110) over i ∈ [ n ] that

where the step (e) makes use of the Cauchy-Schwarz inequality.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction provide a good summary of our contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main limitation lies in the lower bounds where we assume the existence of a sequence of Markov policies that are close enough to the history-dependent behavioral policies.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide rigorous analysis of both the upper and lower bounds in our paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We don't have experimental results in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We don't have experimental results in this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We don't have experimental results in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We don't have experimental results in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We don't have experimental results in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We don't see any violations.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is mostly theoretical, and is not tied to a particular application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We give full credit to the prior work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to estimating linear functionals of the reward function in contextual bandit settings from adaptively collected data. They consider the class of augmented inverse propensity weighted (AIPW) estimators and prove guarantees about the quality of the estimator in terms of the quality of the plugin estimator of the mean reward. Specifically, they characterize the finite sample MSE of the AIPW estimator on adaptively collected data. They quality of the estimator (in terms of low MSE) depends on the quality of the plugin estimator of the mean. They relate online learning of the plugin estimator of the mean reward to online non-parametric regression; they then prove a regret bound, where regret relates to the quality of the plug-in estimator learned online.\n\n***edited score based on rebuttal. See comment below***\n\nSTRENGTHS:\nThis paper's presentation is technically precise and seems mathematically thorough. The author's approach of relating the estimation problem online learning seems interesting and creative.\n\nWEAKNESSES:\n- There is significant missing discussion of relevant related work on finite sample approaches for off policy evaluation in contextual bandits. For example \"Anytime-valid off-policy inference for contextual bandits\" by Waubly-Smith et al., \"Off-Policy Confidence Sequences\" by Karampatziakis et al. and \"Optimal and Adaptive Off-policy Evaluation in Contextual Bandits\" by Wang et al., as some examples. I would recommend the authors compare to these papers both theoretically and ideally experimentally as well.\n\n- The main result the authors present that holds for general plug-in estimators of the mean is the derivation of the MSE upper bound for the AIPW estimator (Theorem 3.2) as a function of the \"regret\" of the plug-in estimator error. Beyond this the authors provide results for the plug-in estimators in tabular data settings (Theorem 3.3) and plug-in estimators that are linear models (Theorem 3.4). I feel this is a limited contribution in terms of the types of plug-in estimators the authors can provide guarantees for. \n\n- The main results presented in this work assume that the behavior has a constant exploration rate (Assumption 1). Other works on statistical inference after adaptive sampling generally can prove results when the exploration can decay.\n\n- The writing is technically precise but extremely dense and often lacks sufficient context. I give explicit examples below:\n\n(-) The final result in section 4 was extremely hard to parse. There is a mention of \"mis-specification\" in the section 4.1 title, but then there is no mention of mis-specification anywhere in the section itself. It is not easy to understand how the result presented in that section is related to mis-specification.\n\n(-) There was little discussion as to why the pertubed IPW estimator was introduced in section 3.1, rather than just starting with the AIPW estimator in 3.2 (since there were no examples discussed of the perturbed IPW estimator that were not some version of an AIPW estimator). Furthermore, if the discussion in 3.1 is kept, the connection between these two estimators presented in 3.1 and 3.2 respectively needs to be made more explicitly. For example, explicitly relating AIPW if for a certain choice of $f$ these two estimators are the same.\n\nQUESTIONS:\n- In line 97, I do not understand why $\\mu$ is called a treatment effect. It is also called the \"reward model\" and appears to be the expected reward function (function of context and action). Why is this called a treatment effect? This is not what is \"commonly referred to\" as a treatment effect in the causal inference literature.\n- In line 107 you state that its assumed the propensities are \"revealed\", does this mean known?\n- Please define << notation used in 117\n\nLIMITATIONS:\n- A severe limitation of this work (especially compared to previous literature) is that they only provide guarantees about the mean squared error of the AIPW estimator and do not provide any approach to constructing confidence intervals for the quantities of interest. I would expect most people interested in using statistical inference methods on adaptively collected data (especially if data is small enough to warrant using a linear plug-in model or be in a tabular data setting) would care solely about estimation and not uncertainty quantification. As a result, the practical utility of this work is very limited. \n\n- There are no simulations demonstrating their approach in practice or comparing to other methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the challenge of estimating a linear functional of the treatment effect from adaptively collected data, commonly found in contextual bandits and causal inference studies. It introduces finite-sample upper bounds for the mean-squared error (MSE) of augmented inverse propensity weighting (AIPW) estimators and proposes a reduction scheme to minimize these bounds. The method is illustrated through three concrete examples. Additionally, the paper establishes a local minimax lower bound, demonstrating the instance-dependent optimality of the AIPW estimator.\n\nSTRENGTHS:\n-\tThe paper extends the non-asymptotic theory of AIPW estimators to adaptively collected data.\n-\tThe paper provides both an upper bound and a local minimax lower bound on the MSE of the off-policy value, which quantifies the similarity between a given target evaluation function $g$ and the treatment effect $\\mu^*$.\n\nWEAKNESSES:\n-\tAlthough the paper is primarily theoretical, it would be helpful if the authors could include some simulation experiments to verify the theoretical results. For example, it would be interesting to see how the regret converges in practice in the examples of Section 3.5.\n-\tIt would be beneficial if the authors could provide more explanations regarding certain definitions. For example, the off-policy value is defined as the expectation of the inner product between $g$ and $\\mu^*$, rather than, e.g., the expectation of $g$ itself. What is the rationale behind this definition? In addition, why is the perturbed IPW estimator considered over the traditional IPW estimator?\n-\tThe paper has generalized the theory developed for i.i.d. data to adaptively collected data and discussed the technical difficulties in Section 3.2. Could the authors compare the results for i.i.d. data with those for adaptively collected data? Is there any efficiency loss when the data is collected adaptively?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nLimitations have been discussed in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper study the off-policy problem in the sequential decision setting with adaptively collected data. The authors propose to use augmented Inverse propensity weighting estimator to estimate the policy value and conduct extensive theoretical analysis on the estimator, including variance and mean square error. Based on the analysis, the authors propose the methods to learn the function estimator in AIPW estimators.\n\nSTRENGTHS:\nThis paper comprehensively analyze the property of the AIPW estimator, including the variance and MSE bound. Therefore, I think this is a theoretically solid paper. And the connection between theory and the method is smooth and well-grounded.\n\nWEAKNESSES:\nI am confused about the claim of \"adaptive\" and \"online learning\" in the title. The two words seems that the decision-maker can adaptively select the action during the decision process. However, it seems that the record (context, actions, outcomes) are passively observed in the problem. So I concern that the paper may be mis-positioned.\n\nTherefore, the technical distinction between the sequential decision setting and static setting is not clearly presented. It seem to be a trivial extention of the traditional policy evaluations problem in the static setting.\n\nQUESTIONS:\nThe formulation of  perturbed IPW estimator in section 3.1 seems different with the doubly robust estimator.  Can the authors provide the connection between the perturbed IPW estimator and the doubly robust estimator in other papers. For example, the doubly robust estimators in [1].\n\n[1] Miroslav Dudik, John Langford, and Lihong Li. 2011. Doubly robust policy evaluation and learning. In International Conference on International Conference on Machine Learning. 1097–1104.\n\nLIMITATIONS:\nI think the discussion of the limitation in the paper is not sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}