Title: Adaptive Linear Estimating Equations
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandit algorithm

statistical inference

adaptively collected data

asymptotic normality
Abstract: Sequential data collection has emerged as a widely adopted technique for enhancing the efficiency of data gathering processes. Despite its advantages, such data collection mechanism often introduces complexities to the statistical inference procedure. For instance, the ordinary least squares (OLS) estimator in an adaptive linear regression model can exhibit non-normal asymptotic behavior, posing challenges for accurate inference and interpretation. In this paper, we propose a general method for constructing debiased estimator which remedies this issue. It makes use of the idea of adaptive linear estimating equations, and we establish theoretical guarantees of asymptotic normality, supplemented by discussions on achieving near-optimal asymptotic variance. A salient feature of our estimator is that in the context of multi-armed bandits,  our estimator retains the non-asymptotic performance of the least squares estimator while obtaining asymptotic normality property. Consequently, this work helps connect two fruitful paradigms of adaptive inference: a) non-asymptotic inference using concentration inequalities  and b) asymptotic inference via asymptotic normality.

Adaptive Linear Estimating Equations

Abstract

Sequential data collection has emerged as a widely adopted technique for enhancing the efficiency of data gathering processes. Despite its advantages, such data collection mechanism often introduces complexities to the statistical inference procedure. For instance, the ordinary least squares (OLS) estimator in an adaptive linear regression model can exhibit non-normal asymptotic behavior, posing challenges for accurate inference and interpretation. In this paper, we propose a general method for constructing debiased estimator which remedies this issue. It makes use of the idea of adaptive linear estimating equations, and we establish theoretical guarantees of asymptotic normality, supplemented by discussions on achieving near-optimal asymptotic variance. A salient feature of our estimator is that in the context of multi-armed bandits, our estimator retains the non-asymptotic performance of the least squares estimator while obtaining asymptotic normality property. Consequently, this work helps connect two fruitful paradigms of adaptive inference: a) non-asymptotic inference using concentration inequalities and b) asymptotic inference via asymptotic normality.

1 Introduction

Adaptive data collection arises as a common practice in various scenarios, with a notable example being the use of (contextual) bandit algorithms. Algorithms like these aid in striking a balance between exploration and exploitation trade-offs within decision-making processes, encompassing domains such as personalized healthcare and web-based services [35, 24, 3, 22]. For instance, in personalized healthcare, the primary objective is to choose the most effective treatment for each patient based on their individual characteristics, such as medical history, genetic profile, and living environment. Bandit algorithms can be used to allocate treatments based on observed response, and the algorithm updates its probability distribution to incorporate new information as patients receive treatment and their response is observed. Over time, the algorithm can learn which treatments are the most effective for different types of patients.

Although the adaptivity in data collection improves the quality of data, the sequential nature (non-iid) of the data makes the inference procedure quite challenging [34, 26, 5, 28, 27, 10, 30, 29]. There is a lengthy literature on the problem of parameter estimation in the adaptive design setting. In a series of work [15, 19, 17], the authors studied the consistency of the least squares estimator for an adaptive

linear model. In a later work, Lai [14] studied the consistency of the least squares estimator in a nonlinear regression model. The collective wisdom of these papers is that, for adaptive data collection methods, standard estimators are consistent under a mild condition on the maximum and minimum eigenvalues of the covariance matrix [19, 14]. In a more recent line of work [1, 2], the authors provide a high probability upper bound on the ℓ 2 -error of the least squares estimator for a linear model. We point out that, while the high probability bounds provide a quantitative understanding of OLS, these results assume a stronger sub-Gaussian assumption on the noise variables.

The problem of inference, i.e. constructing valid confidence intervals, with adaptively collected data is much more delicate. Lai and Wei [19] demonstrated that for a unit root autoregressive model, which is an example of adaptive linear regression models, the least squares estimator doesn't achieve asymptotic normality. Furthermore, the authors showed that for a linear regression model, the least squares estimator is asymptotically normal when the data collection procedure satisfies a stability condition. Concretely, letting x i denote the covariate associated with i -th sample, the authors require

where S n = ∑ n i =1 x i x ⊤ i and { B n } n ≥ 1 is a sequence of non-random positive definite matrices. Unfortunately, in many scenarios, the stability condition (1) is violated [38, 19]. Moreover, in practice, it might be difficult to verify whether the stability condition (1) holds or not. In another line of research [10, 36, 37, 4, 9, 28, 31, 25, 38], the authors assume knowledge of the underlying data collection algorithm and provide asymptotically valid confidence intervals. While this approach offers intervals under a much weaker assumption on the underlying model, full knowledge of the data collection algorithm is often unavailable in practice.

Online debiasing based methods: In order to produce valid statistical inference when the stability condition (1) does not hold, some authors [8, 7, 13] utilize the idea of online debiasing. At a high level, the online debiased estimator reduces bias from an initial estimate (usually the least squares estimate) by adding some correction terms, and the online debiasing procedure does not require the knowledge of the data generating process. Although this procedure guarantees asymptotic reduction of bias to zero, the bias term's convergence rate can be quite slow.

In this work, we consider estimating the unknown parameter in an adaptive linear model by using a set of adaptive linear estimating equations (ALEE). We show that our proposed ALEE estimator achieves asymptotic normality without knowing the exact data collection algorithm while addressing the slowly decaying bias problem in online debiasing procedure.

2 Background and problem set-up

In this section, we provide the background for our problem and set up a few notations. We begin by defining the adaptive data collection mechanism for linear models.

2.1 Adaptive linear model

Suppose a scalar response variable y t is linked to a covariate vector x t ∈ R d at time t via the linear model:

where θ ∗ ∈ R d is the unknown parameter of interest.

In an adaptive linear model, the regressor x t at time t is assumed to be a (unknown) function of the prior data point { x 1 , y 1 , . . . , x t -1 , y t -1 } as well as additional source of randomness that may be present in the data collection process. Formally, we assume there is an increasing sequence of σ -fields {F t } t ≥ 0 such that

For the noise variables { ϵ t } t ≥ 1 appearing in equation (2), we impose the following conditions

for some δ > 0 . The above condition is relatively mild compared to a sub-Gaussian condition.

Examples of adaptive linear model arise in various problems, including multi-armed and contextual bandit problems, dynamical input-output systems, adaptive approximation schemes and time series models. For instance, in the context of the multi-armed bandit problem, the design vector x t is one of the basis vectors { e k } k ∈ [ d ] , representing an arm being pulled, while θ ∗ , y t represent the true mean reward vector and reward at time t , respectively.

2.2 Adaptive linear estimating equations

As we mentioned earlier, the OLS estimator can fail to achieve asymptotic normality due to the instability of the covariance matrix with adaptively collected data. To get around this issue, we consider a different approach ALEE (adaptive linear estimating equations). Namely, we obtain an estimate by solving a system of linear estimating equations with adaptive weights,

Here the weight w t ∈ R d is chosen in a way that w t ∈ F t -1 for t ∈ [ n ] . Let us now try to gain some intuition behind the construction of ALEE. Rewriting equation (4), we have

Notably, the choice of w t ∈ F t -1 makes ∑ n t =1 w t ϵ t the sum of a martingale difference sequence. Our first theorem postulates conditions on the weight vectors { w t } t ≥ 1 such that the right-hand side of (5) converges to normal distribution asymptotically. Throughout the paper, we use the shorthand W t = ( w 1 , . . . , w t ) ⊤ ∈ R t × d , X t = ( x 1 , . . . , x t ) ⊤ ∈ R t × d .

Proposition 2.1. Suppose condition (3) holds and the predictable sequence { w t } 1 ≤ t ≤ n satisfies

Let A w = V w U ⊤ w X n with W n = U w Λ w V ⊤ w being the SVD of W n . Then,

where ̂ σ is any consistent estimator for σ .

Proof. Invoking the second part of the condition (6), we have that Λ w is invertible for large n , and ∥ V w Λ -1 w V ⊤ w -I d ∥ op = o p (1) . Utilizing the expression (5), we have

Invoking the stability condition on the weights { w t } and using the fact that ∑ n t =1 w t ϵ t is a martingale difference sequence, we conclude from martingale central limit theorem [11, Theorem 2.1] that

Combining the last equation with ∥ V w Λ -1 w V ⊤ w -I d ∥ op = o p (1) and using Slutsky's theorem yield

The claim of Proposition 2.1 now follows from Slutsky's theorem.

A few comments regarding the Proposition 2.1 are in order. Straightforward calculation shows

In words, the volume of the confidence region based on (7) is always larger than the confidence region generated by the least squares estimate. Therefore, the ALEE-based inference, which is consistently valid, exhibits a reduced efficiency in cases where both types of confidence regions are valid. Compared with the confidence regions based on OLS, the advantage of the ALEE approach is to provide flexibility in the choice of weights to guarantee the validity of the CLT conditions (6).

Next, note that the matrix A w is asymptotically equivalent to the matrix W ⊤ n X n (see equation (5)) under the stability condition (6). The benefit of this reformulation is that it helps us better understand

efficiency of ALEE compared with the OLS. This has led us to define a notion of affinity between the weights { w t } t ≥ 1 and covariates { x t } t ≥ 1 for better understanding of the efficiency of ALEE and ways to design nearly optimal weights, as it will be clear in the next section.

Finally, it is straightforward to obtain a consistent estimate for σ . For instance, assuming log( λ max ( X ⊤ n X n )) /n a.s. -→ 0 and the noise condition (3), we have

Here, ̂ θ LS refers to the least squares estimate. See [19, Lemma 3] for a detailed proof of equation (9).

3 Main results

In this section, we propose methods to construct weights { w t } t ≥ 1 which satisfy the stability property (6), and study the resulting ALEE. Section 3.1 is devoted to the multi-armed bandit case, Section 3.2 to an autoregressive model, and Section 3.3 to the contextual bandit case. Before delving into details, let us try to understand intuitively how to construct weights that have desirable properties.

The expression (8) reveals that the efficiency of ALEE depends on the projection of the data matrix X n on W n . Thus, the efficiency of the ALEE approach can be measured by the principal angles between the random projections P w in (8) and P x = X n S -1 n X ⊤ n . Accordingly, we define the affinity A ( W n , X n ) of the weights { w t } t ≥ 1 as the cosine of the largest principle angle, or equivalently

as the d -th largest singular value of P w P x . Formally, the above definition captures the cosine of the angle between the two subspaces spanned by the columns of X n and W n , respectively [12]. Good weights { w t } t ≥ 1 are those with relatively large affinity or

3.1 Multi-armed bandits

In the context of the K -arm bandit problem, the Gram matrix has a diagonal structure, which means that we can focus on constructing weights { w t } t ≥ 1 for each coordinate independently. For an arm k ∈ [ K ] and round t ≥ 1 , define

Define the k -th coordinate of the weight w t as

The intuition behind the above construction is as follows. The discussion at near equation (11) indicates that the k -th coordinate of w t should be proportional to x t,k / ( ∑ i ≤ n x 2 i,k ) 1 / 2 . However, the weight w t is required to be predictable, which can only depend on the data points 1 up to time t . Consequently, we approximate the sum ∑ i ≤ n x 2 i,k by the partial sum s t,k in (12). Finally, note that

The logarithmic factors in (13) ensure that the stability conditions (6) hold. In the following theorem, we generalize the above method as a general strategy for constructing weights { w t } t ≥ 1 satisfying the stability condition (6).

4

3

2

1

0

1

2

3

4

Distribution of errors in AR(1)

(0, 1)

OLS

4

3

2

1

0

1

2

3

4

Distribution of errors with -greedy

(0, 1)

OLS

3.1.1 Stable weight construction strategy

Consider a positive decreasing function f ( x ) on the interval [1 , ∞ ) with an increasing derivative f ′ ( x ) . Let f satisfy the conditions: f ′ /f is increasing,

With s 0 ∈ F 0 , we define weight w t,k as

A key condition that ensures the weights { w t,k } t ≥ 1 satisfy the desirable stability property (6) is

For multi-armed bandits, this condition is automatically satisfied when both quantities 1 /s 0 and s 0 /s n,k converge to zero in probability. Putting together the pieces, we have the following result for multi-armed bandits.

Theorem 3.1. Suppose condition (3) holds and 1 /s 0 + s 0 /s n,k = o p (1) for some k ∈ [ K ] . Then, the k -th coordinate ̂ θ ALEE , k , obtained using weights from equation (16) , satisfies

where ̂ σ is a consistent estimate of σ . Equivalently,

The proof of Theorem 3.1 can be found in Section A.1 of the Appendix. A few comments regarding Theorem 3.1 are in order.

First, the above theorem enables us to construct valid CI in the estimation of the mean θ ∗ k for a sub-optimal arm k when employing an asymptotically optimal allocation rule to achieve the optimal regret in [18] with sample size ∑ t ≤ n x t,k ≍ log n , or when using a sub-optimal rule to achieve polylog ( n ) . On the other hand, the classical martingale CLT is applicable to the optimal arm (if unique) under such asymptotically optimal or sub-optimal allocation rules. Consequently, one may obtain a valid CI for the optimal arm from the standard OLS estimate [19]. However, it is important to note that such CIs are not guaranteed for sub-optimal arms.

Next, while Theorem 3.1 holds for any s 0 diverging to infinity but of smaller order than s n,k ( which may depend on k ), the convergence rate of ∑ 1 ≤ t ≤ n w t,k ϵ t to normality is enhanced by choosing a large value for s 0 . In practical terms, it is advisable to choose an s 0 that is slightly smaller than the best-known lower bound for s n,k .

Finally, the choice of function f determines the efficiency of ALEE estimator. For instance, taking function f ( x ) = 1 /x , we obtain an estimator with asymptotic variance of order 1 / { s 0 log 2 ( s n,k /s 0 ) } , which is only better than what one would get using stopping time results by a logarithmic factor. In the next Corollary, an improved choice of f yields near optimal variance up to logarithmic terms.

Corollary 3.2. Consider the same set of assumptions as stated in Theorem 3.1. The ALEE estimator ̂ θ ALEE , k , obtained by using f ( x ) = ( β log β 2) 1 / 2 { x (log e 2 x )(log log e 2 x ) 1+ β } -1 / 2 for any β > 0 satisfies

The proof of this corollary follows directly from Theorem 3.1. For s 0 = log n/ log log n in multiarmed bandits with asymptotically optimal allocations, log( s n,k /s 0 ) = (1 + o (1)) log log s n,k .

3.1.2 Finite sample bounds for ALEE estimators

One may also construct finite sample confidence intervals for each arm via applying concentration bounds. Indeed, for any arm k ∈ K , we have

Following the construction of w t,k ∈ F t -1 , the term ∑ n t =1 w t,k ϵ t is amenable to concentration inequalities if we assume that the noise ϵ t is sub-Gaussian conditioned on F t -1 , i.e.

Corollary 3.3 (Theorem 1 in [1]) . Suppose the sub-Gaussian noise condition (21) is in force. Then for any δ > 0 and λ 0 > 0 , the following bound holds with probability at least 1 -δ

Remark 3.4. In the context of multi-armed bandit, by considering the function f in Corollary 3.2 with β = 1 and Corollary 3.3 with λ 0 = 1 , we derive that with probability at least 1 -δ

provided s 0 > 1 . See Section A.2 of the Appendix for a proof of this argument. Recall that √ s n,k = ( s 0 + ∑ i ≤ n x 2 i,k ) 1 / 2 , the bound is in the same spirit as existing finite sample bounds for the OLS estimator for arm means [1, 21]. In simple terms, the ALEE estimator behaves similarly to the OLS estimator in a non-asymptotic setting while still maintaining asymptotic normality.

3.2 Autoregressive time series

Next, we focus on an autoregressive time series model

where y 0 = 0 . Note that the above model is a special case of the adaptive linear model (2). It is well-known that when θ ∗ ∈ ( -1 , 1) , the time series model (24) satisfies a stability assumption (1). Consequently, one might use the OLS estimate based confidence intervals [19] for θ ∗ . However, when θ ∗ = 1 -also known as the unit root case - stability condition (1) does not hold, and the

least squares estimator is not asymptotically normal [19]. In other words, when θ ∗ = 1 , the least squares based intervals do not provide correct coverage.

In this section, we apply ALEE-based approach to construct confidence intervals that are valid for θ ∗ ∈ [ -1 , 1] . Similar to previous sections, let s 0 ∈ F 0 and denote s t = s 0 + ∑ 1 ≤ i ≤ t y 2 i -1 . Following a construction similar to the last section, we have the following corollary.

Corollary 3.5. Assume the noise variables { ϵ t } t are i.i.d with mean zero, variance σ 2 and subGaussian parameter σ 2 g . Then, for any θ ∗ ∈ [ -1 , 1] , the ALEE estimator, obtained using w t = f ( s t /s 0 ) y t -1 / √ s 0 with function f from Corollary 3.2 and s 0 = n/ log log( n ) , satisfies

The proof of Corollary 3.5 can be found in Section A.3 of the Appendix.

3.3 Contextual bandits

In contextual bandit problems, the task of defining adaptive weights that satisfy the stability con- dition (6) while maintaining a large affinity is challenging. Without loss of generality, we assume that ∥ x t ∥ 2 ≤ 1 . Following the discussion around (11) and using S t as an approximation of S n , we see that a good choice for the weight is w t ≈ S -1 2 t x t . However, it is not all clear at the moment why the above choice produces d -dimensional weights w t satisfying the stability condition (6). It turns out that the success of our construction is based on the variability of certain matrix V t . For a F 0 -measurable d × d symmetric matrix Σ 0 ⪰ I d and t ∈ [ n ] , we define

For t ∈ [ n ] , we define the variability matrix V t as

The variability matrix V t comes up frequently in finite sample analysis of the least squares estimator [16, 19, 2], the generalized linear models with adaptive data [23], and in online optimization [6]; see comments after Theorem 3.6 for a more detailed discussion on the matrix V t . Now, we define weights { w t } t ≥ 1 as

Theorem 3.6. Suppose condition (3) holds and ∥ Σ -1 0 ∥ op + ∥ V n ∥ op = o p (1) . Then, the ALEE estimator ̂ θ ALEE , obtained using the weights { w t } 1 ≤ t ≤ n from (28) , satisfies

The proof of Theorem 3.6 can be found in Section A.4 of the Appendix. In Theorem B.4 in the appendix, we establish the asymptotic normality of a modified version of the ALEE estimator, which has the same asymptotic variance as the one in Theorem 3.6 under the assumption ∥ Σ -1 0 ∥ op = o p (1) . In other words, the modified theorem B.4 does not assume any condition on the ∥ V n ∥ op.

To better convey the idea of our construction, we provide a lemma that may be of independent interest. This lemma applies to weights w t generated by (27) and (28) with general z t .

Lemma 3.7. Let w t be as in (28) with the variability matrix V t in (27) . Then,

For z t ∈ F t -1 , the stability condition (6) holds when max 1 ≤ t ≤ n z ⊤ t V t z t + ∥ V n ∥ op = o p (1) .

Comments on Theorem 3.6 conditions: It is instructive to compare the conditions of Theorem 3.1 and Theorem 3.6. The condition ∥ Σ -1 0 ∥ op = o p (1) is an analogue of the condition 1 /s 0 = o p (1) . The condition ∥ V n ∥ op = o p (1) is a bit more subtle. This condition is an analogue of the condition s 0 /s n,k = o p (1) . Indeed, applying elliptical potential lemma [2, Lemma 4] yields

where S n = ∑ n i =1 x i x ⊤ i is the Gram matrix. We see that for ∥ V n ∥ op = o p (1) , it is necessary that the eigenvalues of S n grow to infinity at a faster rate than the eigenvalues of Σ 0 . Moreover, in the case of dimension d = 1 , the condition ∥ V n ∥ op = o p (1) is equivalent to s 0 /s n,k = o p (1) .

4 Numerical experiments

In this section, we consider three settings: two-armed bandit setting, first order auto-regressive model setting and contextual bandit setting. In two-armed bandit setting, the rewards are generated with same arm mean ( θ ∗ 1 , θ ∗ 2 ) = (0 . 3 , 0 . 3) , and noise is generated from a normal distribution with mean 0 and variance 1 . To collect two-armed bandit data, we use ϵ -Greedy algorithm with decaying exploration rate √ log( t ) /t . The rate is designed to make sure the number of times each armed is pulled has order greater than log( n ) up to time n . In the second setting, we consider the time series model,

where θ ∗ = 1 and noise ϵ t is drawn from a normal distribution with mean 0 and variance 1 . In the contextual bandit setting, we consider the true parameter θ ∗ to be 0 . 3 times the all-one vector. In the initial iterations, a random context x t is generated from a uniform distribution in S d -1 . Then, we apply ϵ -Greedy algorithm to these pre-selected contexts with decaying exploration rate log 2 ( t ) /t . For all of the above three settings, we run 1000 independent replications.

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.6

0.7

0.8

0.9

1.0

Empirical coverage probability

Bandit: lower tail coverage for

*

1

Target Coverage

OLS

Concentration

W-decorrelation

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.6

0.7

0.8

0.9

1.0

Empirical coverage probability

Bandit: upper tail coverage for

*

1

Target Coverage

OLS

Concentration

W-decorrelation

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.1

0.2

0.3

0.4

0.5

0.6

Mean width of CI

Average two-sided CI width for

*

1

OLS

Concentration

W-decorrelation

To analyze the data we collect for these settings, we apply ALEE approach with weights specified in Corollary 3.2, 3.5 and Theorem B.4, respectively. More specifically, in the first two settings, we consider β = 1 in Corollary 3.2. For two-armed bandit example, we set s 0 = e 2 log( n ) , which is known to be a lower bound for s n, 1 . For AR(1) model, we consider s 0 = e 2 n/ log log( n ) . For the contextual bandit example, we consider Σ 0 = log( n ) · I d . In the simulations, we also compare ALEE approach to the normality based confidence interval for OLS estimator [19] (which may be incorrect), the concentration bounds for the OLS estimator based on self-normalized martingale sequence [1], and W-decorrelation [8]. Detailed implementations about these methods can be found in Appendix C.1.

In Figure 2, we display results for two-armed bandit example, providing the empirical coverage plots for the first arm mean θ ∗ 1 as well as average width for two-sided CIs. We observe that CIs based on

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.6

0.7

0.8

0.9

1.0

Empirical coverage probability

AR: lower tail coverage for

*

Target Coverage

OLS

Concentration

W-decorrelation

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.6

0.7

0.8

0.9

1.0

Empirical coverage probability

AR: upper tail coverage for

*

Target Coverage

OLS

Concentration

W-decorrelation

0.75

0.80

0.85

0.90

0.95

Target coverage probability

0.002

0.009

0.016

0.023

Mean width of CI

Average two-sided CI width for

*

OLS

Concentration

W-decorrelation

OLS undercover θ ∗ 1 while other methods provide satisfactory coverage. Notably, from the average CI width plot, we can see that W-decorrelation and concentration methods have relatively large CI widths. On the contrary, ALEE-based CIs achieve target coverage while keeping the width of CIs relatively small.

For AR(1) model, we display the results in Figure 3. For the context bandit example, we consider d = 20 and summarize the empirical coverage probability and the logarithm of the volume of the confidence regions in Table 1, along with corresponding standard deviations. See Appendix C.2 for experiments with dimension d = 10 and d = 50 .

5 Discussion

In this paper, we study the parameter estimation problem in an adaptive linear model. We propose to use ALEE (adaptive linear estimation equations) to obtain point and interval estimates. Our main contribution is to propose an estimator which is asymptotically normal without requiring any stability condition on the sample covariance matrix. Unlike the concentration based confidence regions, our proposed confidence regions allow for heavy tailed noise variables. We demonstrate the utilitity of our method by comparing our method with existing methods.

Our work leaves several questions open for future research. For example, it would be interesting to characterize the variance of the ALEE estimator compared to the best possible variance[13, 20] for d > 1 . It would also be interesting to know if such results can be extended to non-linear adaptive models, e.g., to an adaptive generalized linear model [23]. Furthermore, our paper assumes a fixed dimension d for the problem while letting n →∞ . It would be interesting to explore whether we can allow the dimension to grow with the number of samples at a specific rate.

Acknowledgments

This work was partially supported by the National Science Foundation Grants DMS-2311304, CCF1934924, DMS-2052949 and DMS-2210850.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Online least squares estimation with selfnormalized processes: An application to bandit problems. arXiv preprint arXiv:1102.2670 , 2011.

Deepak Agarwal, Bee-Chung Chen, Pradheep Elango, Nitin Motgi, Seung-Taek Park, Raghu Ramakrishnan, Scott Roy, and Joe Zachariah. Online models for content optimization. Advances in Neural Information Processing Systems , 21, 2008.

Aurélien Bibaut, Maria Dimakopoulou, Nathan Kallus, Antoine Chambaz, and Mark van Der Laan. Post-contextual-bandit inference. Advances in neural information processing systems , 34:28548-28559, 2021.

Jack Bowden and Lorenzo Trippa. Unbiased estimation for response adaptive clinical trials. Statistical methods in medical research , 26(5):2376-2388, 2017.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.

Yash Deshpande, Adel Javanmard, and Mohammad Mehrabi. Online debiasing for adaptively collected high-dimensional data with applications to time series analysis. Journal of the American Statistical Association , pages 1-14, 2021.

Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive linear models. In International Conference on Machine Learning , pages 1194-1203. PMLR, 2018.

Maria Dimakopoulou, Zhimei Ren, and Zhengyuan Zhou. Online multi-armed bandits with adaptive inference. Advances in Neural Information Processing Systems , 34:1939-1951, 2021.

Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences , 118(15):e2014602118, 2021.

Inge S Helland. Central limit theorems for martingales with discrete or continuous time. Scandinavian Journal of Statistics , pages 79-94, 1982.

Ilse CF Ipsen and Carl D Meyer. The angle between complementary subspaces. The American mathematical monthly , 102(10):904-911, 1995.

Koulik Khamaru, Yash Deshpande, Lester Mackey, and Martin J Wainwright. Near-optimal inference in adaptive linear regression. arXiv preprint arXiv:2107.02266 , 2021.

Tze Leung Lai. Asymptotic properties of nonlinear least squares estimates in stochastic regression models. The Annals of Statistics , pages 1917-1930, 1994.

Tze Leung Lai and Herbert Robbins. Adaptive design and stochastic approximation. The annals of Statistics , pages 1196-1221, 1979.

Tze Leung Lai and Herbert Robbins. Adaptive design and stochastic approximation. The annals of Statistics , pages 1196-1221, 1979.

Tze Leung Lai and Herbert Robbins. Consistency and asymptotic efficiency of slope estimates in stochastic approximation schemes. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete , 56(3):329-360, 1981.

Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Tze Leung Lai and Ching Zong Wei. Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics , 10(1):154-166, 1982.

Tor Lattimore. A lower bound for linear and kernel regression with adaptive covariates. In The Thirty Sixth Annual Conference on Learning Theory , pages 2095-2113. PMLR, 2023.

Tor Lattimore and Csaba Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728-737. PMLR, 2017.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Peng Liao, Kristjan Greenewald, Predrag Klasnja, and Susan Murphy. Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 4(1):1-22, 2020.

Licong Lin, Koulik Khamaru, and Martin J Wainwright. Semi-parametric inference based on adaptively collected data. arXiv preprint arXiv:2303.02534 , 2023.

Alexander R Luedtke and Mark J Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics , 44(2):713, 2016.

Seth Neel and Aaron Roth. Mitigating bias in adaptive data gathering via differential privacy. In International Conference on Machine Learning , pages 3720-3729. PMLR, 2018.

Xinkun Nie, Xiaoying Tian, Jonathan Taylor, and James Zou. Why adaptively collected data have negative bias and how to correct for it. In International Conference on Artificial Intelligence and Statistics , pages 1261-1269. PMLR, 2018.

Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. Are sample means in multi-armed bandits positively or negatively biased? Advances in Neural Information Processing Systems , 32, 2019.

Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. On the bias, risk and consistency of sample means in multi-armed bandits. arXiv preprint arXiv:1902.00746 , 2019.

Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11070-11078, 2020.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Min Xu, Tao Qin, and Tie-Yan Liu. Estimation bias in multi-armed bandit algorithms for search advertising. Advances in Neural Information Processing Systems , 26, 2013.

Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning system. Journal of medical Internet research , 19(10):e338, 2017.

Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125-2135, 2021.

Ruohan Zhan, Zhimei Ren, Susan Athey, and Zhengyuan Zhou. Policy learning with adaptively collected data. Management Science , 2023.

Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. Advances in neural information processing systems , 33:9818-9829, 2020.

Appendix

Table of Contents

A Proof

In Theorem 3.1, Corollary 3.3 and Remark 3.4, we deal with an arm with index k ∈ [ K ] . To simplify notations, we drop the subscript k in s t,k , w t,k , x t,k , ̂ θ ALEE , k and θ ∗ k throughout the proof, and use s t , w t , x t , ̂ θ ALEE and θ ∗ , respectively.

A.1 Proof of Theorem 3.1

Condition (17) serves as an important role in proving (18). Therefore, we start our proof by verifying the condition (17). Since function f is a positive decreasing function, we first have

Furthermore, since function f ′ /f is increasing, we have

where inequality ( i ) follows from mean value theorem and the monotonicity of the function f ′ /f . Thus, by assuming 1 /s 0 = o p (1) and s 0 /s n = o p (1) , condition (17) follows directly from equation (32) and equation (33).

By the construction of ALEE estimator, we have

Note that

By the mean value theorem, we have that for t ∈ [ n ] , ξ t ∈ [ s t -1 , s t ]

Therefore, we have

Observe that

Equality ( ii ) follows from condition (17). Consequently, applying Slutsky's theorem yields

Similarly, we can derive

Knowing max 1 ≤ t ≤ n w 2 t = max 1 ≤ t ≤ n f 2 ( s t /s 0 ) x 2 t /s 0 = o p (1) , which is a consequence of equation (17), martingale central limit theorem together with an application of Slutsky's theorem yields

Therefore, equation (19) follows from martingale central limit theorem and Slutsky's theorem.

Remark A.1. Equation (18) sheds light on the asymptotic variance of the ALEE estimator, thereby aiding in the selection of a suitable function f to improve the efficiency of ALEE estimator. On the other hand, equation (19) offers a practical approach to obtaining an asymptotically precise confidence interval.

Remark A.2. Condition (17) is a general requirement that governs equation (18) , and is not specific to bandit problems. However, the difficulty in verifying (17) can vary depending on the problem at hand.

A.2 Proof of Remark 3.4

Corollary 3.3 follows directly from Theorem 1 in [1]. In this section, we provide a proof of Remark 3.4. By considering λ 0 = 1 in Corollary 3.3, we have with probability at least 1 -δ

By the construction of the weights in Corollary 3.2, we have

Lastly, we recall that

Therefore, to complete the proof, it suffices to characterize a lower bound for ∑ 1 ≤ t ≤ n w t x t . By definition, we have

In equation ( i ) , we plug in the expression of function f and hence √ s 0 cancels out. Since x t is either 0 or 1 , inequality ( ii ) follows from the integration of the function h ( x ) = 1 / √ x . Inequality ( iii ) follows from s 0 > 1 . Putting things together, we have

This completes our proof of Remark 3.4.

A.3 Proof of Corollary 3.5

Note that it suffices to verify the following condition (41)

for θ ∗ ∈ [ -1 , 1] in order to complete the proof of Corollary 3.5. The other part of the proof can be adapted from the proof of Theorem 3.1. To simplify notations, we let

Therefore, proving equation (41) is equivalent to showing that T 1 , T 2 , and T 3 converge to zero in probability. We will now demonstrate the convergence of each of these three terms to zero in probability.

T 1 with θ ∗ = 1: To prove T 1 = o p (1) , we make use of a result in [19, Equation 3.23], which is

Observe that

In equation (43), we split the sequence into two parts and set different lower bounds for s t . The major benefit of this step is to help us derive a better choice of s 0 . Now we bound max 1 ≤ t ≤⌊ n 2 / 3 ⌋ y 2 t -1 and

max ⌊ n 2 / 3 ⌋ +1 ≤ t ≤ n y 2 t -1 . Note that

where inequality is derived from [33, Exercise 2.12] and the fact that y i is sub-Gaussian with sub-Gaussian parameter σ 2 g n 2 / 3 for i ≤ ⌊ n 2 / 3 ⌋ . Therefore, we conclude that

By applying the same trick to max ⌊ n 2 / 3 ⌋ +1 ≤ t ≤ n y 2 t -1 , we can derive

Consequently, we have

Hence we have

Equality ( ii ) makes use of equation (42). Combining equation (44) with equations (45) and (46), we conclude that T 1 = o p (1) .

T 1 with θ ∗ = -1 : When θ ∗ = -1 , the proof is essentially the same as the case when θ ∗ = -1 . The only difference lies in the order of ∑ 1 ≤ i ≤ n y 2 i -1 . However, by pairing ϵ 2 t -1 with ϵ 2 t for t ∈ N + , we can arrive at the same result. Specifically, for t ∈ N + , we let ϵ ′ t = ϵ 2 t -ϵ 2 t -1 and define

where y ′ 0 ∆ = 0 and { ϵ ′ t } t ≥ 1 are random variables with mean zero, variance 2 σ 2 and sub-Gaussian parameter 2 σ 2 g . Therefore, applying equation (44) yields

Setting n 0 = ⌊ ( ⌊ n 2 / 3 ⌋ -1) / 2 ⌋ , we have

According to equation (47) and equation (48), we have

which completes the proof of T 1 = o p (1) for the case when θ ∗ = -1 .

T 1 with θ ∗ ∈ ( -1 , 1) : Given θ ∗ ∈ ( -1 , 1) , we observe that y t is a sub-Gaussian random variable with sub-Gaussian parameter σ 2 g 1 -( θ ∗ ) 2 for any t ∈ N + . Therefore, following equation (43), we have

where in the above inequality we use s 0 as a lower bound for s t . By applying [33, Exercise 2.12], we have

leading to the conclusion that T 1 = o p (1) .

T 2 with θ ∗ ∈ [ -1 , 1] : Similar to equation (33), we have

Define g ( x ) = -f ′ ( x ) /f ( x ) and we can compute that

where C is some constant. Doing some calculation yields

Therefore, we have

We note that demonstrating max 1 ≤ t ≤ n y 2 t -1 /s t -1 = o p (1) follows the same approach as the proof of max 1 ≤ t ≤ n y 2 t -1 /s t = o p (1) . Hence, we omit it. To conclude, we show that T 2 = o p (1) for θ ∗ ∈ [ -1 , 1] .

T 3 with θ ∗ ∈ [ -1 , 1] : To prove T 3 = o p (1) , it suffices to verify that

For convenience, in equation (52) we use y t instead of y t -1 . Note that when θ ∗ = 1 or θ ∗ = -1 , we have provided almost sure lower bounds for ∑ 1 ≤ t ≤ n y 2 t in the proof of T 1 = o p (1) . Therefore, equation (52) follows from these lower bounds. To prove equation (52) when θ ∗ ∈ ( -1 , 1) , we begin by rewriting ∑ 1 ≤ t ≤ n y 2 t in quadratic form. Without confusion and loss of generality, we replace θ ∗ by θ , consider Var( ϵ t ) = 1 , and set ε n = ( ϵ 1 , ϵ 2 , . . . , ϵ n ) ⊤ . For t ∈ [ n ] , we have

where a t ∈ R n and a t,j = θ t -j for j ≤ t and a t,j = 0 for j > t . Therefore, ∑ 1 ≤ t ≤ n y 2 t can be written as

where A = ∑ 1 ≤ t ≤ n a t a ⊤ t . Applying Hanson-Wright inequality (e.g. see [32]), we have

where c and K are some universal constants. Observe that

Furthermore, we have

Subsequently, we have

where

Assuming δ ≤ 2 e -c and t = 1 c K 2 | | | A | | | F log( 2 δ ) , we have with probability at least 1 -δ ,

We note that the term on the right hand side of equation (57) has order n . For any ϵ > 0 , consider the following probability

By fixing δ and comparing the order of s 0 with the order of ε ⊤ n A ε n -1 c K 2 | | | A | | | F log( 2 δ ) , we have

Since δ can be arbitrarily small, we conclude that

which completes the proof of T 3 = o p (1) .

A.4 Proof of Theorem 3.6

Note that for any t ≥ 1 , we have

The second part of equation (60) follows from the Sherman-Morrison formula. Let u t = V t z t and we adopt the notation V 0 = I d . By multiplying z t on the right hand side of V t , we have

Therefore, following the definition of u t , we have (1 + z ⊤ t V t -1 z t ) u t = V t -1 z t . Consequently,

By recognizing w t = √ 1 + z ⊤ t V t -1 z t · u t , we come to

What remains now is to verify conditions in (6). Notably, assumption ∥ V n ∥ op = o p (1) implies

Since ∥ Σ -1 0 ∥ op = o p (1) , ∥ V t ∥ op ≤ 1 and ∥ x t ∥ 2 ≤ 1 , we can show

Besides, equation (61) together with equation (64) implies

Thus, it follows that

Combining equations (66) and (63) yields (6). Hence we complete the proof by applying Proposition 2.1.

Remark A.3. The detailed proof of Lemma 3.7 can be found in the proof of Theorem 3.6.

B Generalized Theorem 3.6

In Theorem 3.6, we impose the following condition (67) so that the ALEE estimator with weights specified in equation (28) achieves asymptotic normality:

However, it is typically difficult to directly verify the above condition in practice. To tackle this problem, in this section, we provide a modified version of ALEE estimator which achieves asymptotic normality without requiring condition (67). In this section, we use the same notations Σ t , z t , V t , and w t as defined in equations (26), (27) and (28), respectively. Furthermore, we let λ 1 ≥ . . . ≥ λ n be the eigenvalues of the matrix V -1 n and a 1 , . . . , a n be the corresponding eigenvectors.

At a high level, we construct additional m n vectors { z t } n +1 ≤ t ≤ n + m n so that the minimum eigenvalue of the resulting matrix V -1 n + m n is greater than a pre-specified constant κ n , which satisfies lim n →∞ κ n = ∞ . It is easy to see that by construction (see Algorithm 1), the matrix V n + m n satisfies

Remark B.1. Parameter κ n is set to ensure condition (69) holds. In practice, we set κ n = d log( n ) .

Remark B.2. It's worth mentioning that the number of extra { z t } t>n is a random variable. Therefore, in order to prove a similar asymptotic normality theorem to Theorem 3.6, we have to apply martingale central limit theorem with stopping times [11, Theorem 2.1].

Theorem B.3 (Theorem 2.1 in [11]) . Let { ξ n,k } k ≥ 1 ,n ≥ 1 be an array of random variables defined on a probability space (Ω , F , P ) and let {F n,k } n ≥ 1 ,k ≥ 0 be an array of σ -fields such that ξ n,k is F n,k -measurable and F n,k -1 ⊂ F n,k ⊂ F for each n and k ⩾ 1 . For each n , let k n be a stopping time with respect to {F n,k } k ≥ 0 . Suppose that

Algorithm 1: Modified ALEE estimate

1: Input: { ( x t , y t ) } n t =1 and tuning parameter κ n

3: Initiate t = n and set τ n = 1 / ∥ Σ -1 / 2 0 ∥ op

2: Compute { ( z t , V t , w t ) } n t =1 , { ( λ k , a k ) } d k =1 , and obtain a consistent estimate ̂ σ 2 of σ 2

4: for k = 1 , . . . , d do

5: Compute n k = ⌈ max { κ n -λ k , 0 } · τ n ⌉

6: if n k > 0 then

7:

for

i

= 1

, . . . , n

8:

k

do

Set

t

=

t

+1

9: Simulate ϵ t ∼ N (0 , ̂ σ 2 )

10: Define z t = a k /τ n

11: Compute

12: end for

13: end if

14: end for

15: Obtain ̂ θ ALEE from equation

With this setup, we are now ready to prove the asymptotic normality of ̂ θ ALEE from (68).

Theorem B.4 (Generalized Theorem 3.6) . Suppose condition (3) holds. Then, for any tuning parameters Σ 0 and κ n that satisfy ∥ Σ -1 0 ∥ op = o p (1) and lim n →∞ κ n = ∞ , the ALEE estimator ̂ θ ALEE obtained from equation (68) satisfies

where ̂ σ is a consistent estimator of σ .

Remark B.5. We would like to reiterate that the asymptotic variance of of the modified ALEE estimator obtained from (68) is the same as the one mentioned in Theorem 3.6. Additionally, this modified version does not require the condition ∥ V n ∥ op = o p (1) hold and hence is more applicable in practice with theoretical guarantee.

Proof. Rewriting equation (68), we have

16: Output: ̂ θ ALEE

Therefore, by Cramér-Wold theorem, it suffices to show that for any unit vector v ,

The proof now follows by verifying the conditions (70a)-(70c) of Theorem B.3 with ξ n,k = v ⊤ w k ϵ k . We begin by verifying conditions (70a)-(70c). By Lemma 3.7, we have

Note that

By equation (73) and the fact that ̂ σ 2 is consistent, we have

Combining equations (69), (73), (74) and (75), we conclude

On the other hand, we have

Inequality ( i ) follows from the definition of w t . In inequality ( ii ) , we use the assumption that Σ 0 ⪰ I d and the fact that ∥ z t ∥ 2 ≤ 1 and ∥ V t ∥ op ≤ 1 . The last inequality ( iii ) follows from the definition of z t and the condition that ∥ Σ -1 0 ∥ op = o p (1) . Hence, we can see that

Therefore, we have

Note that condition (70a) holds because { v ⊤ w k ϵ k } k ≥ 1 is a martingale difference sequence by construction. Condition (70b) follows from statement (78). It remains to verify condition (70c). Observe that

Equation ( iv ) follows from condition (3), equation (78) and the fact that ̂ σ 2 is a consistent estimator. Lastly, by applying Slutsky's theorem, we prove that

C Simulation

In this section, we provide additional comparisons among the ALEE method, the OLS, the W-decorrelation [8], and the concentration inequality based bounds [1]. The code can be found at https://github.com/ mufangying/ALEE .

C.1 Simulation details

Throughout our experiments, we utilize ̂ σ 2 from equation (9) as an (consistent) estimate of of σ 2 [19].

OLS: When data are i.i.d, the least squares estimator satisfies the following condition

Therefore, we consider 1 -α confidence region to be

We point out that the above confidence region is not guaranteed to be accurate when the data is collected in an adaptive manner, as will also be highlighted in our experiments.

W-decorrelation: The W-decorrelation method is borrowed from Algorithm 1 in [8]. Specifically, the estimator takes the form

Given a parameter λ , weights { w t } 1 ≤ t ≤ n are set as follows

Following the recommendations from the paper [8], in order to set λ appropriately, we first run the bandit algorithm or time series with N replications and record the corresponding minimum eigenvalues { λ min ( S (1) n ) , . . . , λ min ( S ( N ) n ) } . We choose λ to be the 0.1-quantile of { λ min ( S (1) n ) , . . . , λ min ( S ( N ) n ) } . Finally, we obtain a 1 -α confidence region for θ ∗ as

where W = ( w 1 , . . . , w n ) ⊤ .

Concentration based on self-normalized martingales: We consider [1, Theorem 1] for a single coordinate in two-armed bandit problem and AR(1) model. For contextual bandits, we apply [1, Theorem 2]. Applying concentration bounds requires a sub-Gaussian parameter, for which we use ̂ σ from equation (9) as an estimate. We point out that this estimate of the sub-Gaussian parameter is conservative, as the sub-Gaussian parameter of a sub-Gaussian random variable is always lower bounded by its variance [33, Chapter 2]. This variance estimate is accurate for Gaussian noise random variables.

For one dimensional examples, we have that for any λ > 0 , with probability at least 1 -α :

In two-armed bandit problem, x t is simply x t, 1 for θ ∗ 1 or x t, 2 for θ ∗ 2 . Here we consider λ = 1 .

For the contextual bandit examples, we apply Theorem 2 from [1], and set S = d ; we set a small value of λ = 0 . 01 to mimic the performance of an OLS estimators. Specifically, we utilize the following 1 -α confidence region

√

C.2 Tables for contextual bandits

In all the contextual bandit simulations, we consider noises that are generated from a centered Poisson distribution (i.e. Poisson (1) -1 ). We would like to highlight that the centered Poisson random variable is not sub-Gaussian. Therefore, it is important to note that concentration inequality-based bounds [1] may not be guaranteed to work. In the simulations of this section, we set the number of samples as n = 1000 , and the tables below show results over 1000 replications. The tables below clearly show that the average log-volume of the confidence regions are smallest for ALEE among methods which yield valid confidence regions (empirical coverage is more than the target coverage). The volume of the confidence region obtained from the OLS estimate is the smallest, but they under-cover the true parameter. The confidence regions for ALEE are obtained from Theorem B.4 with Σ 0 = log( n ) · I d and κ n = d log( n ) .

C.3 Asymptotic normality with centered Poisson noise variables

4

3

2

1

0

1

2

3

4

Distribution of errors in AR(1)

(0, 1)

OLS

4

3

2

1

0

1

2

3

4

Distribution of errors with -greedy

(0, 1)

OLS

Peer reviews:
comments: SUMMARY:
The authors propose a general method for constructing debiased estimator called Adaptive Linear Estimating Equations (ALEE) estimator, which achieves asymptotic normality even in sequential data collection.

To obtain valid statistical inference, the online debiasing concept is used. The online debiasing procedure guarantees asymptotic reduction of bias to zero, but the convergence speed is slow. However, the ALEE estimator solves the slowly decreasing bias problem.

In this paper, the stable weights of ALEE are proved through equations for three cases: multi-arm bandits, autoregressive time series, and contextual bandits. A comparison of the ALEE method confirms that it performs better than the previous method.

STRENGTHS:
- ALEE provides point and interval estimation based on the central limit theorem, which enables stable estimation. It also prove the stability condition of ALEE through formulas for stable weights in three cases (Multi-arm bandits, Autoregressive time series, and Contextual bandits), which increases the efficiency of ALEE. Based on this stability and efficiency, we expect that it can be applied to various models.

- ALEE's training algorithm has fast convergence. The authors provide theoretical performance guarantees and demonstrate ALEE's effectiveness on distributed time series forecasting problems with several examples.

WEAKNESSES:
- In the numerical experiments, the results are only shown using parameter values of 0.3 and 1 for the two-armed bandit setting and the contextual bandit setting. It would be better if used the various values for the parameters.

- Lack of explanation for Table 1.

QUESTIONS:
Table 1: OLEE typo (ALEE), and Table 1 seems to lack explanation.

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper considers the problem of least squares when the data is collected sequentially. It proposes a form of weighted least squares where the weights are designed to lead to estimates that are asymptotically normal and nearly optimal variance. The appropriate weights are derived for the multi-arm bandit, autoregressive, and context bandit settings. Experimental results on some toy datasets confirm the theory.

STRENGTHS:
1. The proposed estimator is simple and nearly efficient.
2. Therefore, I think it would be useful for practitioners.
3. The problem is well-motivated.
I am not familiar with the literature on this problem, so I cannot comment on originality.

WEAKNESSES:
1. The presentation is somewhat confusing at times. The matrix $A$ from Section 2 does not appear in Section 3. The general construction strategy in Section 3.1 does not seem to be applied in Section 3.3 and it's not clear why. 
2. The examples used in the experiments are all very simple. It would strengthen the work to, for example, vary the number of arms/dimension.

QUESTIONS:
Please see above.

LIMITATIONS:
The paper is largely theoretical, so I think it's fine that the authors don't include a broader impacts section. However, I think the authors should include something about the limitations of the current work (see previous sections).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes an estimator (ALEE) for adaptively collected data generated from adaptive linear models, describes its construction such that asymptotic normality holds for practically relevant examples, and demonstrates its desirable properties in numerical experiments.

STRENGTHS:
I enjoy reading this paper. It reads well. 

- The problem of inference for adaptive data is practically relevant
- Theoretical guarantees assume somewhat weaker assumptions than previous works
- The proposed method provides an improvement over other approaches, at least in the numerical experiments shown

WEAKNESSES:
I don't think this submission lacks anything for a NeurIPS paper

QUESTIONS:
- One novelty of the paper is that Eq (3) is weaker than sub-Gaussian, yet all experiments are for Gaussian noise. I would be more convinced if the numerical simulations shows the same desirable properties for non-Gaussian noise.

- Perhaps refer to Fig 1 from the main text?

Some minor typos I find:

- Refs 14&15 are identical
- l51 - _debaising_ → debiasing
- l98 - _Slutsy's theorem_ → Slutsky's theorem

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21

comments: SUMMARY:
This paper introduces a general method for constructing debiased estimator within the context of sequential data collection. The proposed methodology is applied explicitly to multi-arm bandits, autoregressive time series, and contextual bandits. Experiments are conducted in these three domains to verify the applicability and effectiveness.

STRENGTHS:
1.	The proposed method is able to achieve asymptotic normality without knowing the data collection algorithm and can obtain a faster convergence rate of the bias term.
2.	Pointwise and interval estimates can therefore be generated.

WEAKNESSES:
The experiment section only contains synthetic results. Considering the broad application of sequential data collection, it would greatly enhance the study if the authors could validate their framework using real-world datasets. This could provide more practical insight into the effectiveness of the proposed method.

QUESTIONS:
In Figures 2 and 3, what do “lower tail coverage” and “upper tail coverage” represent?

LIMITATIONS:
Limitations are not mentioned, and the potential negative societal impact is not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26